| Title: | Stochastic Frontier Analysis |
|---|---|
| Description: | Maximum likelihood estimation of stochastic frontier models, in which observed output falls short of an unobserved production frontier by a one-sided inefficiency term in addition to symmetric noise, following Aigner, Lovell and Schmidt (1977) <doi:10.1016/0304-4076(77)90052-5> and Meeusen and van den Broeck (1977) <doi:10.2307/2525757>. Cross-sectional models cover seventeen noise and inefficiency distribution pairs, with heteroskedasticity permitted in either error component. Panel models include the true fixed and random effects estimators of Greene (2005) <doi:10.1016/j.jeconom.2004.05.003> and the four-component model of Colombi, Kumbhakar, Martini and Vittadini (2014) <doi:10.1007/s11123-014-0386-y>. Further entry points address zero inefficiency, latent classes, two-tier frontiers, sample selection, nonparametric frontiers, dependence between the error components, and endogenous regressors after Amsler, Prokhorov and Schmidt (2016) <doi:10.1016/j.jeconom.2015.06.013>. Technical efficiency prediction, marginal effects, and specification tests for wrong skewness, for the assumed inefficiency distribution and for heteroskedasticity are provided. |
| Authors: | David Bernstein [aut, cre] (ORCID: <https://orcid.org/0000-0002-2267-5741>), Christopher Parmeter [aut], Alexander Stead [aut] |
| Maintainer: | David Bernstein <[email protected]> |
| License: | GPL (>= 2) |
| Version: | 1.2.1 |
| Built: | 2026-10-14 22:55:10 UTC |
| Source: | https://github.com/davidhbernstein/sfa |
Maximum likelihood estimation of stochastic frontier models, in which observed output falls short of an unobserved production frontier by a one-sided inefficiency term in addition to symmetric noise, following Aigner, Lovell and Schmidt (1977) <doi:10.1016/0304-4076(77)90052-5> and Meeusen and van den Broeck (1977) <doi:10.2307/2525757>. Cross-sectional models cover seventeen noise and inefficiency distribution pairs, with heteroskedasticity permitted in either error component. Panel models include the true fixed and random effects estimators of Greene (2005) <doi:10.1016/j.jeconom.2004.05.003> and the four-component model of Colombi, Kumbhakar, Martini and Vittadini (2014) <doi:10.1007/s11123-014-0386-y>. Further entry points address zero inefficiency, latent classes, two-tier frontiers, sample selection, nonparametric frontiers, dependence between the error components, and endogenous regressors after Amsler, Prokhorov and Schmidt (2016) <doi:10.1016/j.jeconom.2015.06.013>. Technical efficiency prediction, marginal effects, and specification tests for wrong skewness, for the assumed inefficiency distribution and for heteroskedasticity are provided.
The DESCRIPTION file:
| Package: | sfa |
| Version: | 1.2.1 |
| Date: | 2026-09-07 |
| Title: | Stochastic Frontier Analysis |
| Type: | Package |
| Authors@R: | c(person("David", "Bernstein", email = "[email protected]", role = c("aut", "cre"), comment = c(ORCID = "0000-0002-2267-5741")), person("Christopher", "Parmeter", role = c("aut")), person("Alexander", "Stead", role = c("aut"))) |
| Maintainer: | David Bernstein <[email protected]> |
| Description: | Maximum likelihood estimation of stochastic frontier models, in which observed output falls short of an unobserved production frontier by a one-sided inefficiency term in addition to symmetric noise, following Aigner, Lovell and Schmidt (1977) <doi:10.1016/0304-4076(77)90052-5> and Meeusen and van den Broeck (1977) <doi:10.2307/2525757>. Cross-sectional models cover seventeen noise and inefficiency distribution pairs, with heteroskedasticity permitted in either error component. Panel models include the true fixed and random effects estimators of Greene (2005) <doi:10.1016/j.jeconom.2004.05.003> and the four-component model of Colombi, Kumbhakar, Martini and Vittadini (2014) <doi:10.1007/s11123-014-0386-y>. Further entry points address zero inefficiency, latent classes, two-tier frontiers, sample selection, nonparametric frontiers, dependence between the error components, and endogenous regressors after Amsler, Prokhorov and Schmidt (2016) <doi:10.1016/j.jeconom.2015.06.013>. Technical efficiency prediction, marginal effects, and specification tests for wrong skewness, for the assumed inefficiency distribution and for heteroskedasticity are provided. |
| Suggests: | knitr, rmarkdown, testthat, sn, np, DEA, sandwich, lmtest, texreg, pbapply, gsl |
| Imports: | pso, numDeriv, plm, minqa, randtoolbox, mnormt, truncnorm, mvtnorm, Formula, methods, parallel, graphics, grDevices |
| Depends: | R (>= 4.0.0) |
| License: | GPL (>= 2) |
| Language: | en-US |
| URL: | https://www.davidharrybernstein.com/software, https://github.com/davidhbernstein/sfa |
| BugReports: | https://github.com/davidhbernstein/sfa/issues |
| LazyLoad: | yes |
| NeedsCompilation: | no |
| VignetteBuilder: | knitr |
| Config/testthat/edition: | 3 |
| Config/testthat/parallel: | true |
| Config/pak/sysreqs: | make |
| Repository: | https://davidhbernstein.r-universe.dev |
| Date/Publication: | 2026-10-14 22:03:28 UTC |
| RemoteUrl: | https://github.com/davidhbernstein/sfa |
| RemoteRef: | HEAD |
| RemoteSha: | 6aaaa4cd388d0c8f78b572696571d6ce28b0e9bc |
| Author: | David Bernstein [aut, cre] (ORCID: <https://orcid.org/0000-0002-2267-5741>), Christopher Parmeter [aut], Alexander Stead [aut] |
Index of help topics:
calibrate_c Fixed robustness parameter by weight matching
copsfm Stochastic Frontier with Dependent, and
Optionally Skewed, Error Components
cw_test Chen and Wang (2012) centered-residuals moment
test
data_gen_cs Generate Cross-Sectional Data for Stochastic
Frontier Analysis
data_gen_p Generate Panel Data for Stochastic Frontier
Analysis
density_weights Density-power weights from a robust frontier
fit
efficiency Technical Efficiency Scores
efficiency_ci Confidence Intervals for Individual
Inefficiency and Efficiency
endogeneity_test Were the regressors endogenous after all?
esfm Stochastic frontier tolerating the wrong
skewness
FinnishElec Expenditure and output of Finnish electricity
distribution firms
gof_test Goodness-of-fit test for the assumed
inefficiency distribution
hscore Hyvarinen score for a Normal-Half-Normal
frontier fit
hscore_select Select the robustness tuning parameter by
Hyvarinen score
Indian Paddy farming in Aurepalle, India, 1975-76 to
1984-85
inefficiency_test Test whether there is any technical
inefficiency
influence_sfa Which observations are driving this frontier
fit?
ivsfm Stochastic Frontier Models with Endogenous
Regressors
lcsfm Latent Class Stochastic Frontier Models
lcsfm_homogeneity Test a latent class frontier against
homogeneity
marginal_effects Marginal Effects of the Variance Determinants
on Inefficiency
meanefficiency Model-Implied Mean Efficiency
moment_range What skewness and excess kurtosis can a pair of
distributions produce?
npsfm Nonparametric Stochastic Frontier Models
panel89 Costs of U.S. commercial banks, 1989
pcomposed The distribution of the composed error
pcomposed_model Composed-error distribution function for any
cross-sectional model
PL80_MVTN Pitt and Lee (1981) Model III: the multivariate
truncated normal panel likelihood
predict.sfareg Predictions, fitted values and residuals for
stochastic frontier fits
print.sfareg Print method for "sfareg" objects
psfm Panel Stochastic Frontier Models
psfm_bootstrap Parametric Bootstrap for 'psfm()' Panel
Stochastic Frontier Models
selsfm Stochastic Frontier Models with Sample
Selection or Technology Choice
sfa_diagnostics Optimizer Diagnostics for Stochastic Frontier
Fits
sfa-package Stochastic Frontier Analysis
sfa-wrongskew The wrong-skewness suite: what to do when the
residuals lean the wrong way
sfareg-methods Extractor methods for "sfareg" objects
sfm Cross-Sectional Stochastic Frontier Models
sfma Average over inefficiency distributions instead
of choosing one
simulation_se Simulation Standard Errors for Simulated-ML
Fits
skewness_decomp Where the skewness of the composed error comes
from
skewness_test Formal tests for wrong skewness in the OLS
residuals
spec_test Is this pair of distributions defensible?
summary.sfareg Summary method for "sfareg" objects
TIC Choosing between non-nested stochastic frontier
specifications
tt_boundary_report Is a collapsed two-tier noise scale the
maximum, or did the optimizer stop short?
ttsfm Two-Tier Stochastic Frontier Model
uhet_test Test whether technical inefficiency depends on
firm characteristics
USUtilities U.S. fossil-fuel steam electric utilities,
1986-1997
zsfm Zero-Inefficiency Stochastic Frontier Models
https://www.davidharrybernstein.com/software
https://github.com/davidhbernstein/sfa
## Simple application of the generalized true random effects estimator. library(sfa) data_trial <- data_gen_p(t=10,N=100, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) psfm(formula = y_gtre ~ x1 + x2, model_name = "GTRE", data = data_trial, individual = "name", PSopt = FALSE)## Simple application of the generalized true random effects estimator. library(sfa) data_trial <- data_gen_p(t=10,N=100, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) psfm(formula = y_gtre ~ x1 + x2, model_name = "GTRE", data = data_trial, individual = "name", PSopt = FALSE)
Chooses the robustness tuning parameter so that a residual a given number of
scale units from the reference point retains a common share of its clean-data
influence, comparably across the robust criteria. This is the transparent
alternative to the data-driven selection of hscore_select.
calibrate_c(sigma_v, sigma_u, method = c("mlqe", "psi", "mdpd", "all"), target = 0.10, k = 3, range = c(1e-4, 0.6))calibrate_c(sigma_v, sigma_u, method = c("mlqe", "psi", "mdpd", "all"), target = 0.10, k = 3, range = c(1e-4, 0.6))
sigma_v, sigma_u
|
Positive scalars at which to calibrate, typically the estimates from a preliminary fit or the values of a simulation design. |
method |
Robust criterion, or |
target |
Share of clean-data influence the reference residual should
retain. Default |
k |
Reference residual in scale units. Default |
range |
Search interval for the tuning parameter. |
The calibration matches the norm of the observationwise estimating-equation
contribution in the intercept, and
directions. Matching on the intercept direction alone would be vacuous: there
the criteria share the density-power factor , so it would force
the same tuning value for all of them by construction. They differ through the
Fisher-consistency correction, which enters the scale directions.
Note that is the scale parameter of
the composed density, not the standard deviation of the composed error, which is
. The argument k is measured
in the former.
The ratio is not monotone, and there can be two answers. For the
Fisher-consistency-corrected criteria the influence ratio falls below the target
and is then pulled back up again by the integral correction, so it crosses the
target twice. At , the /MDPD
solutions are and . The larger is returned:
it is the calibration reported by Bernstein, Parmeter and Wright, and it lies on
the far side of the dip, so the target down-weighting holds there rather than
being passed through on the way. Both are in the "roots" attribute; the
non-monotonicity is reported rather than resolved silently. MLqE
has a single solution, .
"psi" and "mdpd" always calibrate to the same value: the two
objectives differ by the factor , which is constant in the parameters
and cancels from the ratio being matched.
A named numeric vector of calibrated tuning parameters, carrying an
attribute "roots" that lists every solution found for each criterion.
Bernstein, D.H., Parmeter, C.F. and Wright, I.A. (2026). On Robust Estimation of the Stochastic Frontier Model. Working paper.
hscore_select, density_weights
## the calibration used in Bernstein, Parmeter and Wright calibrate_c(sigma_v = 0.30, sigma_u = 0.60, method = "all")## the calibration used in Bernstein, Parmeter and Wright calibrate_c(sigma_v = 0.30, sigma_u = 0.60, method = "all")
Fits a cross-sectional stochastic frontier in which the noise and the
inefficiency may be dependent, with the dependence carried by a
copula, and in which the noise itself may be skewed. Every other model
in the package assumes independence and a symmetric .
copsfm(formula, data, copula = c("gaussian", "fgm", "frank", "clayton", "clayton90", "clayton180", "clayton270", "gumbel", "gumbel90", "gumbel180", "gumbel270", "joe", "joe90", "joe180", "joe270", "independent"), inefdec = TRUE, udist = c("hnormal", "exponential"), vdist = c("normal", "logistic", "glogistic"), n_nodes = 128, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)copsfm(formula, data, copula = c("gaussian", "fgm", "frank", "clayton", "clayton90", "clayton180", "clayton270", "gumbel", "gumbel90", "gumbel180", "gumbel270", "joe", "joe90", "joe180", "joe270", "independent"), inefdec = TRUE, udist = c("hnormal", "exponential"), vdist = c("normal", "logistic", "glogistic"), n_nodes = 128, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)
formula |
Two-sided formula for the frontier, |
||||||||||||||||||||||||||||
data |
A |
||||||||||||||||||||||||||||
copula |
The dependence family. See ‘Choosing a family’ below.
Clayton, Gumbel and Joe carry only positive dependence, which is a
real restriction: nothing rules out a negative association between noise and
inefficiency, and against negatively dependent data those families can only
report the independence boundary. The
|
||||||||||||||||||||||||||||
inefdec |
|
||||||||||||||||||||||||||||
udist |
Marginal distribution of the inefficiency |
||||||||||||||||||||||||||||
vdist |
Marginal distribution of the noise
|
||||||||||||||||||||||||||||
n_nodes |
Gauss–Legendre nodes for the integral over |
||||||||||||||||||||||||||||
keep_objective |
Logical. Retain the likelihood closure on the fitted object as |
||||||||||||||||||||||||||||
maxit.bobyqa, maxit.psoptim, maxit.optim, start_val, PSopt, optHessian, Method, verbose, rand.psoptim
|
As in |
In brief:
The frontier slopes behave normally. The dependence parameter does
not: it needs several thousand observations, and at its
sign can come out wrong, taking with it (0.57 against a
truth of 1 in the design below).
Below a crossover between and , plain
sfm estimates with a lower RMSE than the true
copula does.
Prefer copula = "frank" or "clayton". The other families
often return the independence bound on data generated from themselves, and
copsfm warns when you pick one.
(vdist = "glogistic") is consistent but noisy,
and below about it can return its upper bound.
Keep n_nodes at 64 or more. At 32 the estimates are biased.
Each point is measured in the subsections below.
The package's taxonomy already writes the general joint density as
with every existing specification setting . This function
relaxes that, and lets the two marginals vary. Composing over
,
a one-dimensional integral evaluated by Gauss–Legendre quadrature on
rather than by simulation.
Measured against the closed-form
normal/half-normal density at independence, the largest absolute error in
is 1.3e-2 at 32 nodes, 2.5e-5 at 64, and 8.2e-14 at 128. Sixty-four
looks adequate and is not: 1e-5 per observation is 0.02 in a log-likelihood over
2000 points, which is the scale at which competing modes are compared.
Node counts, measured in paired runs. Forty datasets sent to every node count,
so the differences are the quadrature and not the draw. Against the 128-node default,
the mean difference in is +0.0002 at 256 nodes () and
+0.0046 at 64 () – neither distinguishable – but -0.0783 at 32
(, ), with per-dataset differences as large as 0.52. So
the quadrature has converged by 64, and 32 must not be used as a fast option: it is
biased, not merely noisier. For scale, 's own sampling standard
deviation is 0.385 at , which dwarfs every node-count effect here.
Bonanno and Domma (2022) argue that the “wrong skewness” anomaly is a specification failure rather than a small-sample accident. The third central moment of the composed error is
so it depends on the skew of and on the dependence between and
, not only on the skew of . In a model where is symmetric
and independent of the first term is the only one, and a positive
residual skew has nowhere to go but . vdist =
"glogistic" gives it somewhere else to go.
Their four specifications are four (copula, vdist) pairs, all with
udist = "exponential":
| their label | call | |
| (I,S) | copula = "independent", vdist = "logistic" |
independence, symmetric |
| (I,A) | copula = "independent", vdist = "glogistic" |
independence, skewed |
| (D,S) | copula = "fgm", vdist = "logistic" |
dependence, symmetric |
| (D,A) | copula = "fgm", vdist = "glogistic" |
dependence, skewed
|
The paper derives the composed density in closed form for
this case, as a sum of four Gauss hypergeometric terms – Theorem 1 for a
production frontier and Theorem 2 for a cost frontier. This function computes
it by quadrature instead, and the two agree to about in
across , and both orientations; the
check is in test-copsfm.R. Note that the paper's summary moment formulas
are not all consistent with its own appendix: Eq. (9)'s dependence term is half
what the appendix's implies, and Eq. (10)'s leading
term is where an exponential's third central moment is
. Simulation follows the appendix. The density theorems –
which are what is implemented here – are correct.
The same two slips appear in Eq. (10) and Eq. (11) of the earlier production
paper (Bonanno, De Giovanni and Domma 2017), and the corrected expressions have
since been derived and pinned. Writing , the covariance is – twice the published value – so for a production
frontier
and
The published dependence bracket carries an extra on top of the factor of two. Both corrections rest on the FGM
copula's exact factorization , whose two generalized-logistic weights are
and ; those are verified to in
test-wrongskew-suite.R. skewness_decomp computes the
three pieces for any fit.
is consistent but noisy, and below about it
can return its upper bound. Measured over 12 replications at each size, data
generated at , ,
and (for FGM) :
| copula | |
median |
IQR | on the bound | RMSE |
| independent | 1000 | 2.54 | [1.49, 10.21] | 3/12 | 1.88 |
| independent | 4000 | 3.05 | [2.38, 3.31] | 0/12 | 1.20 |
| independent | 16000 | 2.77 | [2.21, 2.90] | 0/12 | 0.48 |
| FGM | 1000 | 3.53 | [1.92, 4.61] | 1/12 | 4.27 |
| FGM | 4000 | 2.78 | [2.63, 3.90] | 0/12 | 1.25 |
| FGM | 16000 | 2.63 | [2.38, 2.95] | 0/12 | 0.61 |
and are unaffected throughout – their medians
stay within 2% of the truth at every size, as does by
(0.689 against 0.7). So the cost of a poorly determined
falls on alone.
On an individual sample the profile can be monotone all the way to the
bound, and the boundary point can genuinely attain a higher likelihood than
the truth. On one such sample at the profile log-likelihood rose
from -2627.5 at to -2622.5 at (the
truth attaining -2626.3), with falling from 0.73 to 0.38 along
the way – trades off against the dependence parameter and the
intercept, not against the two scales. Read a boundary as
“the noise is skewed, direction known, magnitude not”, and compare
against vdist = "logistic" rather than reporting the number.
The families differ in two things that matter: whether they admit negative dependence, and where they concentrate it. Gaussian and Frank spread association evenly and have no tail dependence; Clayton concentrates it in the lower tail and Gumbel and Joe in the upper; FGM can only express weak association at all.
Against that, weigh what the next subsection establishes: the dependence
parameter is estimated very imprecisely. On a Gaussian design at
its sampling standard deviation is 0.385 against a truth of 0.5.
Offering fifteen families does not make that better, and it makes one thing
worse – with fifteen candidates, some family will attain the highest
likelihood on independent data by chance. On a sample generated with
independent and , the fitted log-likelihoods across
families span only about 0.8, and the best of them reports
for Gumbel. Treat a family comparison as descriptive, not as evidence of a
dependence structure, and prefer a family chosen for a reason to one chosen by
its likelihood.
The dependence parameter needs a large sample. This is the thing to know
before using it. The frontier slopes behave normally, but does not.
Most of these families do not recover their own dependence parameter, and
copsfm warns when you pick one. A correct density does not imply an
estimable parameter. Measured on 25 samples per family, generated from that
family at and refitted with it:
| family | mean estimate (truth) | fits on the independence bound |
"frank" |
5.43 (5) | 0% |
"clayton" |
2.24 (2) | 0% |
"gumbel" |
1.29 (2) | 36% |
"joe" |
1.52 (2) | 40% |
"clayton270" |
0.23 (2) | 56% |
"gumbel90" |
1.26 (2) | 60% |
On data generated from a Gumbel copula with Spearman at
, every family – including the true one – returns the
independence boundary, and their log-likelihoods differ by less than 0.04. The
likelihood is close to flat in the dependence parameter. This is a property of
the model rather than of the implementation: each density is checked against the
second mixed partial of its own CDF, and each sampler against the family\'s
theoretical Spearman correlation. The remaining rotations were not measured, and
warn that they were not, rather than implying either outcome.
Prefer "frank" or "clayton", and treat an estimate from any
other family as exploratory.
On a Gaussian-copula design with true , ,
:
|
|
|
|
| 2000 | -0.247 | 0.571 | 0.275 |
| 8000 | 0.528 | 0.921 | 0.379 |
| 20000 | 0.571 | 0.971 | 0.389 |
It is consistent – the estimates converge on the truth – but at
the sign can come out wrong. The cause is joint identification rather than a
bad optimiser: with and the scales held at their true values the
likelihood peaks exactly at , so is well identified
conditionally; it is the trade-off against ,
and the intercept that defeats it in moderate samples, and the fitted point
genuinely attains a higher likelihood than the truth.
Treat as informative only in large samples, and compare against
an independent fit (sfm) before reading anything into it.
And below a few hundred observations, the independent fit is often the better
estimate – even when the dependence is real. Fitting the true copula gives a nearly
unbiased but very noisy ; ignoring the copula gives a precise but
biased one. RMSE of (truth 1.0) on Gaussian-copula data, 25
replications:
| design | |
true copula | independence |
|
400 | 0.431 | 0.240 |
|
1000 | 0.151 | 0.220 |
|
400 | 0.251 | 0.196 |
|
1000 | 0.175 | 0.203 |
The crossover sits between and . Below it the variance
cost of estimating jointly with exceeds the bias it
removes, so plain sfm wins on RMSE; above it copsfm wins. The
independence bias does not shrink with (about 22% of , since
it is misspecification rather than noise), while the copula fit's RMSE falls as it
should.
Only densities that could be verified are offered. Each is checked in
test-copsfm.R: it integrates to 1 over the unit square, and it returns
exactly 1 at the independence parameter. Frank, Clayton, Gumbel and Joe, whose
densities concentrate in a corner where the integral check is loose, are also
checked against the second mixed partial of their own CDF, a test a wrong
formula cannot pass. The rotations are those same densities reflected.
An object of class "sfareg", with copula, copula_par
(NA when copula = "independent"), udist, vdist and
alpha_v alongside the usual components. jlms is
and exp_u_hat is
, both by the same quadrature the likelihood
used; efficiency is exp(-jlms), which is what this function has
always returned under that name and is not the Battese–Coelli
predictor.
Smith, M. D. (2008). Stochastic frontier models with dependent error components. The Econometrics Journal, 11(1), 172–192.
Bonanno, G. and Domma, F. (2022). Analytical derivations of new specifications for stochastic frontiers with applications. Mathematics, 10(20), 3876.
sfm for the independent case.
set.seed(4) ## n_nodes = 64 rather than the default 128: the quadrature is converged by 64 ## (see Details), and this is the slowest entry point in the package. n <- 600 x1 <- rnorm(n); x2 <- rnorm(n) z1 <- rnorm(n); z2 <- 0.6 * z1 + sqrt(1 - 0.6^2) * rnorm(n) y <- 0.5 + 0.8 * x1 - 0.4 * x2 + 0.4 * z1 - qnorm((1 + pnorm(z2)) / 2) dat <- data.frame(y = y, x1 = x1, x2 = x2) fit <- copsfm(y ~ x1 + x2, data = dat, n_nodes = 64) fit$outset.seed(4) ## n_nodes = 64 rather than the default 128: the quadrature is converged by 64 ## (see Details), and this is the slowest entry point in the package. n <- 600 x1 <- rnorm(n); x2 <- rnorm(n) z1 <- rnorm(n); z2 <- 0.6 * z1 + sqrt(1 - 0.6^2) * rnorm(n) y <- 0.5 + 0.8 * x1 - 0.4 * x2 + 0.4 * z1 - qnorm((1 + pnorm(z2)) / 2) dat <- data.frame(y = y, x1 = x1, x2 = x2) fit <- copsfm(y ~ x1 + x2, data = dat, n_nodes = 64) fit$out
Tests the distributional assumption on the composed error using the characteristic function of the centred residuals, without ever evaluating the composed-error density or distribution function.
cw_test(x, model_name = NULL, tau = 1, type = c("cosine", "sine"), data = NULL, n_nodes = 256L)cw_test(x, model_name = NULL, tau = 1, type = c("cosine", "sine"), data = NULL, n_nodes = 256L)
x |
An |
model_name |
The distributional pair under the null. Required when
|
tau |
Frequency at which the characteristic function is compared. Chen and Wang suggest a value around 1, and a single frequency: combining several oversizes the test (see Details). |
type |
|
data |
The data the model was fitted to, when |
n_nodes |
Quadrature nodes for the theoretical moments. |
Why this test rather than gof_test. It never touches the
composed-error PDF or CDF. It compares the empirical characteristic function
of the centred residuals with its theoretical value, and that follows from the
characteristic functions of the two components separately, which are in
closed form for many pairs whose convolution is not – the normal-gamma model
being the standard example of a pair with no closed-form composed density.
Centring is what makes it work. A frontier's intercept is not
identified separately from , so the least-squares intercept
estimates rather than . Centring the residuals
cancels that inconsistent intercept outright, which is why the test is valid
off an ordinary regression and does not require maximum likelihood. The
variance then has to correct for two nuisances – estimating the scale
parameters, and the centring itself. Omitting the second understates the
variance and oversizes the test.
Use one frequency, not several. Measured here over 500 replications at a nominal 5%, half-normal data:
| test | |
|
|
|
| cosine | 500 | 0.070 | 0.054 | 0.050 |
| cosine | 2000 | 0.046 | 0.044 | 0.050 |
| sine | 500 | 0.092 | 0.064 | 0.042 |
| sine | 2000 | 0.068 | 0.062 | 0.064 |
Against exponential inefficiency the cosine test's power at is
0.231 at and 0.952 at , against 0.108 and 0.706
for the sine test. Both of Chen and Wang's conclusions reproduce: the cosine
test is the less sensitive to the choice of , and the more powerful.
Passing several frequencies at once gives rejection rates of 0.09 to 0.15 at a
nominal 5%, because the cosine moments at nearby frequencies are close to
collinear and the covariance matrix is then near-singular; the authors report
the same for their combined .
The moment estimator solves the second- and third-moment equations, which for
"NHN" and "NE" is the closed form of the paper's equation (32)
and is the same inversion sfm(estimator = "cols") uses. A wrongly
signed third moment leaves it with no admissible solution, and that is an
error rather than a number: see skewness_test.
An object of class "cw_test": the statistic D, its degrees of
freedom and p-value, the moment estimates of and
, and the frequencies used.
Chen, Y.-T. and Wang, H.-J. (2012). Centered-residuals-based moment estimator and test for stochastic frontier models. Econometric Reviews 31(6), 625–653.
gof_test, spec_test, skewness_test
set.seed(4) n <- 500 X <- matrix(rnorm(n * 3), n, 3) y <- X %*% c(1, 1, 1) + rnorm(n, 0, 1) - abs(rnorm(n, 0, 1)) e <- residuals(lm(y ~ X)) cw_test(e, "NHN") # true pair cw_test(e, "NE") # wrong inefficiency distributionset.seed(4) n <- 500 X <- matrix(rnorm(n * 3), n, 3) y <- X %*% c(1, 1, 1) + rnorm(n, 0, 1) - abs(rnorm(n, 0, 1)) e <- residuals(lm(y ~ X)) cw_test(e, "NHN") # true pair cw_test(e, "NE") # wrong inefficiency distribution
data_gen_cs generates simulated cross-sectional data based on the stochastic frontier model, allowing for different distributional assumptions for the one-sided technical inefficiency error term () and the two-sided idiosyncratic error term (). The model has the general form:
where and represents inefficiency. All variants are produced so that the user can select those that they want.
data_gen_cs(N, rand, sig_u, sig_v, cons, beta1, beta2, a, mu, sig_w = sig_u, shape_g = 2, m_nak = 1, mu_ln = -0.5, k_w = 1.5, lam_tsl = 1.5)data_gen_cs(N, rand, sig_u, sig_v, cons, beta1, beta2, a, mu, sig_w = sig_u, shape_g = 2, m_nak = 1, mu_ln = -0.5, k_w = 1.5, lam_tsl = 1.5)
N |
A single integer specifying the number of observations (cross-sectional units). |
rand |
A single integer to set the seed for the random number generator, ensuring reproducibility. |
sig_u |
The standard deviation parameter ( |
sig_v |
The standard deviation parameter ( |
cons |
The value of the constant term (intercept) in the model. |
beta1 |
The coefficient for the |
beta2 |
The coefficient for the |
a |
The degrees of freedom parameter for the t half-t distribution ( |
mu |
The mean parameter ( |
sig_w |
The standard deviation/scale parameter ( |
shape_g |
Shape of the gamma inefficiency draw behind |
m_nak |
Nakagami shape |
mu_ln |
Log-scale mean of the lognormal inefficiency draw behind |
k_w |
Weibull shape behind |
lam_tsl |
Skew parameter |
The function simulates two explanatory variables, and , as transformations of uniform random variables.
The function generates several different frontier models by combining various distributions for and :
** Distributions (Inefficiency):** Half-Normal (HN), Truncated Normal (TN), Half-T (HT), Half-Cauchy (HC), Exponential (E), Half-Uniform (HU).
** Distributions (Idiosyncratic):** Normal (N), t, Cauchy (C).
**Specific Model Outputs (y_pcs variants):**
y_pcs: Normal-Half-Normal (N-HN): , .
y_pcs_z: N-HN with Heteroskedastic : , where is a uniform variable.
y_pcs_t: T-Half T (T-HT): , .
y_pcs_st: the column that actually matches sfm's THT. y_pcs_t above draws its two components as two independent rt() variates, which shares the degrees of freedom but not the mixing variable, so the composed error is not skew-t and THT cannot recover it. y_pcs_st uses the single common of Tancredi (2002), i.e. . y_pcs_t is retained unchanged only for backward compatibility.
y_pcs_thn: Student's t–half-normal, matching sfm's tHN: and , drawn independently. Heavy-tailed noise with a conventional one-sided term – distinct from both y_pcs_t and y_pcs_st, in which the inefficiency is heavy-tailed too.
y_pcs_tn: Normal-Truncated Normal (N-TN): , on .
y_pcs_e: Normal-Exponential (N-E): , , where .
y_pcs_c: Cauchy-Half Cauchy (C-HC): , .
y_pcs_u: Normal-Half Uniform (N-HU): , .
y_pcs_w: Normal + Cauchy - Half-Normal: , . This introduces a composite term.
**Note:** The rtruncnorm function is required for y_pcs_tn and loads with the package. In isolation it could be loaded by using library(truncnorm).
A data frame containing observations with the following columns:
name |
Individual identifier (simply |
cons |
The constant term value. |
x1 |
Simulated explanatory variable |
x2 |
Simulated explanatory variable |
u, uz, u_t, u_c, u_e, u_u, u_tn, u_st, u_thn
|
The simulated one-sided error terms under different distributions. |
u_r |
Rayleigh one-sided error term, for |
v, v_t, v_c, v_st, v_thn
|
The simulated two-sided error terms under different distributions. |
lam_st |
The common Gamma(a/2, a/2) scale-mixing variable behind |
y_pcs, y_pcs_t, y_pcs_st, y_pcs_thn, y_pcs_e, y_pcs_ez, y_pcs_c, y_pcs_u, y_pcs_z, y_pcs_w, y_pcs_tn
|
The dependent variable |
y_pcs_r |
Normal-Rayleigh dependent variable, for |
z |
The auxiliary variable used for heteroskedasticity in |
uz_e |
Exponential one-sided error term with heteroskedastic scale, for |
w_tt, w_tt_hn, wz_hn
|
The second one-sided error component ( |
zp |
A second auxiliary variable used for heteroskedasticity of |
y_ttne |
Homoskedastic two-tier dependent variable ( |
y_tthn |
Homoskedastic two-tier dependent variable (both one-sided components half-normal), for |
y_tthn_z |
Heteroskedastic two-tier dependent variable ( |
eff_ind, eff_ind_z
|
Indicator (1 = efficient, |
prob_z_true |
The true heteroskedastic "efficient regime" probability used to draw |
y_zisf |
Zero-inefficiency dependent variable with constant efficient-regime probability, for |
y_zisf_z |
Zero-inefficiency dependent variable with |
u_g, y_pcs_g
|
Gamma one-sided error term and the corresponding dependent variable, for |
u_nak, y_pcs_nak
|
Nakagami one-sided error term and dependent variable, for |
u_ge, y_pcs_ge
|
Generalized-exponential one-sided error term and dependent variable, for |
u_ln, y_pcs_ln
|
Lognormal one-sided error term and dependent variable, for |
u_w, y_pcs_wb
|
Weibull one-sided error term and dependent variable, for |
con |
A constant column set to 1, potentially for use in estimation. |
David H. Bernstein
data_gen_p for the panel generator, and sfm, zsfm and ttsfm for the estimators these columns are built for.
rnorm, runif, rt, rexp, rcauchy, rtruncnorm (if available).
# Generate 100 observations of SFA data data_sfa <- data_gen_cs( N = 100, rand = 123, sig_u = 0.5, sig_v = 0.2, cons = 5, beta1 = 1.5, beta2 = 2.0, a = 5, # degrees of freedom for T/Half-T mu = 0.1 # mean for Truncated Normal ) # Display the first few rows of the generated data head(data_sfa) # Example of a Normal-Half-Normal SFA model data summary(data_sfa$y_pcs) plot(density(data_sfa$y_pcs))# Generate 100 observations of SFA data data_sfa <- data_gen_cs( N = 100, rand = 123, sig_u = 0.5, sig_v = 0.2, cons = 5, beta1 = 1.5, beta2 = 2.0, a = 5, # degrees of freedom for T/Half-T mu = 0.1 # mean for Truncated Normal ) # Display the first few rows of the generated data head(data_sfa) # Example of a Normal-Half-Normal SFA model data summary(data_sfa$y_pcs) plot(density(data_sfa$y_pcs))
data_gen_p generates simulated panel data for estimating various panel stochastic frontier models, including the Generalized True Random Effects (GTRE), True Random Effects (TRE), Pooled Cross-Section (PCS), and True Fixed Effects (TFE) models. The function returns the data as a pdata.frame. All variants are produced so that the user can select those that they want.
data_gen_p(t, N, rand, sig_u, sig_v, sig_r, sig_h, cons, tau = 0.5, mu = 0, beta1, beta2, eta = 0.1, b_k90 = 0.05, c_k90 = 0.01, d_k90 = 0.05, e_k90 = -0.005, rho_mvtn = 0.5, delta_fd = mu, fd_draw = c("folded", "truncated"))data_gen_p(t, N, rand, sig_u, sig_v, sig_r, sig_h, cons, tau = 0.5, mu = 0, beta1, beta2, eta = 0.1, b_k90 = 0.05, c_k90 = 0.01, d_k90 = 0.05, e_k90 = -0.005, rho_mvtn = 0.5, delta_fd = mu, fd_draw = c("folded", "truncated"))
t |
The number of time periods. |
N |
The number of individuals. |
rand |
A seed for the random number generator to ensure reproducibility. |
sig_u |
The standard deviation ( |
sig_v |
The standard deviation ( |
sig_r |
The standard deviation ( |
sig_h |
The standard deviation ( |
cons |
The constant term ( |
tau |
The dependence parameter ( |
mu |
The pre-truncation mean ( |
delta_fd |
The coefficient |
fd_draw |
How |
beta1 |
The coefficient for the |
beta2 |
The coefficient for the |
eta |
The decay-rate parameter ( |
b_k90, c_k90
|
Coefficients of the Kumbhakar (1990) inefficiency path |
rho_mvtn |
Within-firm correlation of inefficiency across periods for the |
d_k90, e_k90
|
Coefficients of the modified Kumbhakar path |
A pdata.frame object with observations, containing the following columns:
name Individual identifier.
year Time period identifier.
cons The constant term used in the data generation.
x1, x2 Explanatory variables generated from a log-uniform distribution.
x1_w, x2_w Explanatory variables with dependence parameter and linkage with , used for the TFE model.
u, v, r, h The generated error and individual effect components.
y_gtre, y_tre, y_pcs, y_tfe Output variables for the Production Frontier models, including the constant.
y_gtre_nc, y_tre_nc, y_pcs_nc Output variables for the Production Frontier models, excluding the constant.
c_gtre, c_tre, c_pcs, c_tfe Output variables for the Cost Frontier models, including the constant.
c_gtre_nc, c_tre_nc, c_pcs_nc Output variables for the Cost Frontier models, excluding the constant.
y_fd Output variable for the first difference model (see Wang and Ho, 2010).
x_fd Explanatory variable for the y_fd model.
u_fd_star, z_fd, r_fd, u_fd Components used to generate y_fd.
u_gtre, z_gtre, zp_gtre, y_gtre_z,y_gtre_zz,y_tre_z Variables for models with heteroskedastic inefficiency (. Variables for models with heteroskedastic persistent inefficiency (.
u_inv, y_ssfe Time-invariant firm-level inefficiency (, constant within individual across ) and the corresponding output variable, for psfm's "SSFE" and "PL80".
u_bc92, y_bc92 Time-varying inefficiency following a Battese-Coelli (1992) decay structure () and the corresponding output variable, for psfm's "BC92".
B_k90, u_k90, y_k1990 The Kumbhakar (1990) inefficiency path (controlled by b_k90, c_k90), the resulting inefficiency , and the corresponding output variable, for psfm's "K1990".
B_k90m, u_k90m, y_k1990m The modified Kumbhakar path (controlled by d_k90, e_k90), the resulting inefficiency, and the corresponding output variable, for psfm's "K1990modified".
h_z The heteroskedastic persistent inefficiency expanded to one value per observation (constant within individual), used to build y_gtre_zz.
The data is generated based on standard Stochastic Frontier Analysis (SFA) formulations, primarily for a **Production Frontier** where the one-sided error component is subtracted:
y_gtre: GTRE model:
y_tre: TRE model:
y_pcs: PCS model:
y_tfe: TFE model:
y_gtre_z: GTRE with Heteroskedastic : .
y_gtre_zz: GTRE with Heteroskedastic : and .
For **Cost Frontier** models, the one-sided error component is added (e.g., c_gtre).
The error terms are generated as:
(individual two-sided effect)
(individual one-sided effect)
(two-sided noise)
(one-sided inefficiency)
The First-Difference estimation model (y_fd) uses a variation where and , with time-invariant: folded normal by default, truncated normal under fd_draw = "truncated" (see fd_draw).
A pdata.frame object containing observations suitable for Stochastic Frontier Analysis (SFA).
David H. Bernstein
Chen, Y., Schmidt, P., & Wang, H. (2014). Consistent estimation of the fixed effects stochastic frontier model. Journal of Econometrics, 181(2), 65-76.
Filippini, M., & Greene, W. H. (2016). Persistent and transient productive inefficiency: a maximum simulated likelihood approach. Journal of Productivity Analysis, 45, 187-196.
Wang, H., & Ho, C. M. (2010). Estimating fixed-effect panel stochastic frontier models by model transformation. Journal of Econometrics, 157(2), 286-296.
data_gen_cs for the cross-sectional generator, and psfm for the estimators these columns are built for.
library(sfa) # Generate a dataset data_trial <- data_gen_p(t=10, N=100, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, tau = 0.5, mu= 0.5, beta1 = 0.5, beta2 = 0.5) # See the first few rows head(data_trial)library(sfa) # Generate a dataset data_trial <- data_gen_p(t=10, N=100, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, tau = 0.5, mu= 0.5, beta1 = 0.5, beta2 = 0.5) # See the first few rows head(data_trial)
Returns the weight each observation receives in the estimating equation of a
maximum -likelihood or minimum density-power divergence fit. The weight
is the fitted density raised to the power , normalised so the largest is
one. It is formed as rather than ,
so an observation whose density underflows the natural scale still receives its
own weight and keeps its place in the ordering.
density_weights(object, sigma_v = NULL, sigma_u = NULL, c = NULL, normalize = TRUE, log = FALSE)density_weights(object, sigma_v = NULL, sigma_u = NULL, c = NULL, normalize = TRUE, log = FALSE)
object |
A fitted frontier model, or a numeric vector of composed residuals. |
sigma_v, sigma_u
|
Scale parameters. Required when |
c |
Robustness tuning parameter. Taken from the fit when available. |
normalize |
Logical; scale so the maximum weight is one. Default
|
log |
Logical; return |
A numeric vector of weights in , one per observation, or of
log weights in when log = TRUE.
The weight is a function of the fitted density at the observation, so it responds to observations lying far from the fitted surface. It has no purchase on an observation that is wrong in a regressor and, because the fitted surface bends toward it, ends up close to that surface. A mis-recorded input can therefore be highly influential and still receive a weight near one. This is not a defect of the implementation: the bounded influence of these estimators is bounded with respect to response contamination, conditional on a fixed or bounded design. Pair this with a leverage diagnostic, and with case deletion, before concluding that an observation is or is not driving a result.
Bernstein, D.H., Parmeter, C.F. and Wright, I.A. (2026). On Robust Estimation of the Stochastic Frontier Model. Working paper.
set.seed(1) e <- rnorm(200, 0, 0.3) - abs(rnorm(200, 0, 0.6)) w <- density_weights(e, sigma_v = 0.3, sigma_u = 0.6, c = 0.217) summary(w) head(order(w), 10) # the observations the estimator discounts most ## Rank on the log scale when the fit is sharply peaked: there the smallest ## weights are far below the smallest positive double and cannot be told ## apart once exponentiated. lw <- density_weights(e, sigma_v = 0.02, sigma_u = 0.9, c = 0.217, log = TRUE) head(order(lw), 10)set.seed(1) e <- rnorm(200, 0, 0.3) - abs(rnorm(200, 0, 0.6)) w <- density_weights(e, sigma_v = 0.3, sigma_u = 0.6, c = 0.217) summary(w) head(order(w), 10) # the observations the estimator discounts most ## Rank on the log scale when the fit is sharply peaked: there the smallest ## weights are far below the smallest positive double and cannot be told ## apart once exponentiated. lw <- density_weights(e, sigma_v = 0.02, sigma_u = 0.9, c = 0.217, log = TRUE) head(order(lw), 10)
Extracts observation-specific technical efficiency from a fitted frontier, choosing among the three usual point predictors and between the two scales the dependent variable may be on.
efficiency(object, type = c("bc", "jlms", "mode"), logDepVar = TRUE, newdata = NULL)efficiency(object, type = c("bc", "jlms", "mode"), logDepVar = TRUE, newdata = NULL)
object |
An |
type |
Which predictor of |
logDepVar |
|
newdata |
Data to rebuild the fitted frontier from, needed only when
|
Which predictor. The three differ in what they report about the same
posterior, not in how well they are estimated.
by Jensen's inequality, so "bc" is
never below "jlms"; the gap widens as the posterior spreads out.
"mode" is the only one that can equal exactly 1, and it does so for every
observation whose posterior mean is negative – typically much of the efficient
tail. That is a property of the predictor rather than a defect: where the
posterior mean of is negative the single most likely value really is the
boundary. It is available only for the models whose posterior is a truncated
normal ("NHN", "NHN_Z", "NE", "NTN").
Which scale. The package elsewhere assumes throughout that the
dependent variable is logged, which is the usual case and makes a
proportional shortfall. logDepVar = FALSE instead forms
for a production frontier, and for a cost frontier,
where , cost efficiency :
minimum over actual cost, the level-scale counterpart of . The
orientation is read from the fit's inefdec, including when it was passed
as a variable. That is a ratio of two estimated quantities and is
less well behaved than the log version: it is undefined where the fitted
frontier is zero and negative where exceeds it, both of which happen
near zero output. A warning is issued if any score comes back negative or
non-finite, since silence there would be the real failure.
A numeric vector of efficiency scores, one per observation used in the fit.
efficiency_ci for interval estimates of the same quantity.
set.seed(2) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) y <- 6 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.8)) d <- data.frame(y = y, x1 = x1, x2 = x2) fit <- sfm(y ~ x1 + x2, model_name = "NHN", data = d) summary(efficiency(fit)) # Battese-Coelli, log scale summary(efficiency(fit, type = "jlms")) # never above the above summary(efficiency(fit, type = "mode")) # reaches exactly 1 summary(efficiency(fit, logDepVar = FALSE, newdata = d))set.seed(2) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) y <- 6 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.8)) d <- data.frame(y = y, x1 = x1, x2 = x2) fit <- sfm(y ~ x1 + x2, model_name = "NHN", data = d) summary(efficiency(fit)) # Battese-Coelli, log scale summary(efficiency(fit, type = "jlms")) # never above the above summary(efficiency(fit, type = "mode")) # reaches exactly 1 summary(efficiency(fit, logDepVar = FALSE, newdata = d))
Horrace and Schmidt (1996) intervals for the inefficiency term of a
fitted stochastic frontier model, and the implied intervals for technical
efficiency .
efficiency_ci(object, level = 0.95, type = c("both", "u", "te"))efficiency_ci(object, level = 0.95, type = c("both", "u", "te"))
object |
an object of class |
level |
coverage of the interval, a single number strictly between 0
and 1. Defaults to |
type |
which intervals to return: |
Conditional on the fitted parameters, is normal with mean
mu_star and standard deviation sigma_star, truncated below at
zero – the same posterior the Jondrow et al. (1982) and Battese and Coelli
(1988) point predictors average over. Inverting it at and
gives the bounds in closed form, so the interval costs no
estimation beyond the fit itself. The efficiency bounds follow by monotonicity,
with the endpoints swapped, since is decreasing in .
These intervals condition on the estimated parameters. They describe
where sits given and a known frontier, and make no
allowance for sampling error in the slopes or the variance parameters. Horrace
and Schmidt are explicit about this, and it is why the intervals do not narrow
as the sample grows: they measure the irreducible difficulty of splitting a
single residual into noise and inefficiency, not estimation uncertainty. Read a
wide interval as a warning against reading much into that unit's rank.
Available for the models whose posterior really is a truncated normal:
"NHN", "NHN_Z", "NE" and "NTN". Every other
model_name has a posterior of a different shape, for which these
formulas do not hold; the function reports that rather than returning a
misleading number.
A data frame with one row per observation, carrying u_lower,
u_hat, u_upper and/or te_lower, te_hat,
te_upper according to type, and a "level" attribute.
u_hat is the Jondrow et al. (1982) posterior mean and te_hat the
Battese and Coelli (1988) score – exactly the values the fit already reports,
taken from the fitted object rather than recomputed.
Horrace, W. C. and Schmidt, P. (1996). Confidence statements for efficiency estimates from stochastic frontier models. Journal of Productivity Analysis, 7, 257–282.
Jondrow, J., Lovell, C. A. K., Materov, I. S. and Schmidt, P. (1982). On the estimation of technical inefficiency in the stochastic frontier production function model. Journal of Econometrics, 19, 233–238.
Battese, G. E. and Coelli, T. J. (1988). Prediction of firm-level technical efficiencies with a generalized frontier production function and panel data. Journal of Econometrics, 38, 387–399.
dat <- data_gen_cs(N = 200, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs ~ x1 + x2, data = dat, model_name = "NHN") ci <- efficiency_ci(fit, level = 0.90) head(ci) ## How much of an efficiency ranking is real? Compare the spread of the point ## predictions against the width of a single unit's interval. diff(range(ci$te_hat)) median(ci$te_upper - ci$te_lower)dat <- data_gen_cs(N = 200, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs ~ x1 + x2, data = dat, model_name = "NHN") ci <- efficiency_ci(fit, level = 0.90) head(ci) ## How much of an efficiency ranking is real? Compare the spread of the point ## predictions against the width of a single unit's interval. diff(range(ci$te_hat)) median(ci$te_upper - ci$te_lower)
A Wald test of on an ivsfm fit – the hypothesis that the noise is uncorrelated with the reduced-form errors of the endogenous regressors, so that the endogeneity correction was not needed and a plain sfm fit would have been consistent. This is the Durbin–Wu–Hausman idea carried into the stochastic frontier, and is Equation (25) of Hou, Ramalho and Roseta-Palma (2025).
endogeneity_test(object, level = 0.05)endogeneity_test(object, level = 0.05)
object |
A fit from |
level |
Size used for the reported verdict. |
The statistic. With the vector of correlations between and the reduced-form errors,
the number of endogenous regressors. With a single endogenous regressor it is exactly the squared z-ratio already printed for rho_ in the fit.
Why this is an ordinary chi-square. lives in the open unit ball, so is an interior point of the parameter space. That is the opposite of the situation inefficiency_test faces, where sits on a boundary and the null distribution is a chi-bar-square mixture. No mixture is needed here.
Standard errors for . ivsfm parameterises the correlations as , which keeps them inside the unit ball for any real . Before version 1.2.0 the standard errors of were reported as NA on the ground that this map has a non-diagonal Jacobian. It does, but the Jacobian is short –
that is – so by the delta method. At the null , which is what makes the test well calibrated there. ivsfm now reports these standard errors and carries the full as $vcov_rho, which is what the joint test needs.
Size and power. Measured in this package over 1,000 replications per cell on the design of Hou, Ramalho and Roseta-Palma's section 4.1.1, with and varying. The block is size; the rest is power. Monte Carlo standard errors at 1,000 replications are 0.0095, 0.0069 and 0.0031 at the three levels.
|
|
10% | 5% | 1% | |
| 0.0 | 500 | 0.101 | 0.051 | 0.014 | 0.999 |
| 0.0 | 1000 | 0.098 | 0.051 | 0.006 | 1.008 |
| 0.0 | 2000 | 0.100 | 0.053 | 0.009 | 1.009 |
| 0.0 | 4000 | 0.092 | 0.043 | 0.004 | 1.045 |
| 0.1 | 500 | 0.422 | 0.298 | 0.116 | 1.027 |
| 0.1 | 1000 | 0.629 | 0.492 | 0.270 | 0.991 |
| 0.1 | 2000 | 0.860 | 0.787 | 0.578 | 0.983 |
| 0.1 | 4000 | 0.984 | 0.969 | 0.897 | 0.970 |
| 0.2 | 500 | 0.879 | 0.807 | 0.611 | 0.976 |
| 0.2 | 1000 | 0.986 | 0.976 | 0.902 | 0.951 |
| 0.2 | 2000 | 1.000 | 1.000 | 0.998 | 0.971 |
| 0.2 | 4000 | 1.000 | 1.000 | 1.000 | 0.974 |
The test holds its size at every sample size tried, and power rises monotonically to one at both alternatives. The last column is the diagnostic behind that: it is the mean reported standard error of divided by the actual sampling standard deviation of across replications. The statistic is , so a rejection rate away from nominal comes either from itself or from the variance, and only the second would be a defect here. At 0.95 to 1.05 throughout, the variance estimate is sound – which is what separates this test from uhet_test, where the same ratio is 0.93 and the test is conservative in consequence.
At the paper's own design (, ) the larger alternatives it reports are saturated: power is 1.000 at every level at both and . The and rows are included precisely because a column of ones says nothing about how power builds. Replication scripts: spec_tests/hou2025_table2.R and spec_tests/hou2025_endog_size.R.
Reading a non-rejection. Failing to reject is not evidence for exogeneity. is identified through the instruments, so weak instruments make imprecise and the test powerless; the print method says so rather than letting the p-value speak alone.
An object of class c("sfa_endog_test", "htest") carrying statistic (the Wald ), parameter (its degrees of freedom, the number of endogenous regressors), p.value, the estimated rho with its standard errors and z-ratios, and reject.
Hou, Z., Ramalho, J. J. S. and Roseta-Palma, C. (2025). Dealing with endogeneity in stochastic frontier models: A comparative assessment of estimators. Energy Economics 151, 108922.
Amsler, C., Prokhorov, A. and Schmidt, P. (2016). Endogeneity in stochastic frontier models. Journal of Econometrics 190(2), 280-288.
Kutlu, L. (2010). Battese-Coelli estimator with endogenous regressors. Economics Letters 109(2), 79-81.
ivsfm for the fit, inefficiency_test for the boundary-null case, spec_test and moment_range for the distributional assumptions.
set.seed(9) n <- 600 z <- rnorm(n); x1 <- rnorm(n); eps <- rnorm(n) x2 <- z + eps v <- 0.6 * eps + sqrt(1 - 0.36) * rnorm(n) d <- data.frame(y = 0.5 * x1 + 0.5 * x2 + v - abs(rnorm(n)), x1 = x1, x2 = x2, z = z) fit <- ivsfm(y ~ x1 + x2, endogenous = ~ x2, instruments = ~ z, data = d, model_name = "IVCF") endogeneity_test(fit)set.seed(9) n <- 600 z <- rnorm(n); x1 <- rnorm(n); eps <- rnorm(n) x2 <- z + eps v <- 0.6 * eps + sqrt(1 - 0.36) * rnorm(n) d <- data.frame(y = 0.5 * x1 + 0.5 * x2 + v - abs(rnorm(n)), x1 = x1, x2 = x2, z = z) fit <- ivsfm(y ~ x1 + x2, endogenous = ~ x2, instruments = ~ z, data = d, model_name = "IVCF") endogeneity_test(fit)
Fits the extended stochastic frontier model of Hafner, Manner and Simar
(2018), in which one parameter carries both the scale of
inefficiency (its size) and the direction of skewness (its sign), so a
wrongly skewed sample gives a well-defined non-degenerate fit instead of
collapsing.
esfm(formula, data, dist = c("halfnormal", "exponential"), inefdec = TRUE, start = NULL) symmetry_test(object)esfm(formula, data, dist = c("halfnormal", "exponential"), inefdec = TRUE, start = NULL) symmetry_test(object)
formula, data
|
Model and data, as for |
dist |
Base density for the inefficiency term. |
inefdec |
|
start |
Optional numeric start, |
object |
An |
The classical frontier implies a negatively skewed composed error. When the
sample skewness comes out positive – common at small or large
, and a small-sample accident rather than evidence
against the model – the MLE collapses: is exactly zero, every
firm is fully efficient, and the fit is uninformative. That is the Type I
failure skewness_test reports.
This model keeps the frontier and widens the distribution of . For
, has the classical density with scale
; for that density is mirrored about zero,
shifted right by and truncated to , giving
negative skewness for and so positive skewness for the composed
error. The truncation constant is fixed, not estimated: it is chosen
so whichever sign takes, which keeps the
model a one-parameter scale family and avoids the identification difficulties
of bounded-inefficiency models. for the half-normal
and for the exponential.
The classical model is nested at , and
is an interior point rather than a boundary. So the likelihood ratio
test of – no inefficiency, symmetric errors – is an
ordinary , not the chi-bar-square mixture that
inefficiency_test needs for the classical model.
What it does on a wrongly skewed sample. On a 50-observation draw with
positive residual skewness, sfm(model_name = "NHN") returns
and mean efficiency 0.99976 – every firm on the
frontier. esfm() returns with slopes still
near their true values.
Size of the LR test, 2000 replications against the paper's 100,000, on
its design (, ):
|
50 | 100 | 200 | 500 | 1000 |
| this package, 5% | 0.178 | 0.082 | 0.054 | 0.061 | 0.058 |
| paper, 5% | 0.152 | 0.076 | 0.061 | 0.054 | 0.053 |
The test is badly oversized below and usable from ,
which is the paper's own conclusion.
Efficiency bias, and a caveat. The paper reports smaller bias in mean
efficiency than the classical model even when the population skewness has the
correct sign. That reproduces when inefficiency is not small: at
, the bias in is
-0.0001 at against the classical model's +0.021. But at
, where 22% of samples are wrongly skewed, this model
overshoots the other way: bias -0.040 against the classical +0.036, and it is
the larger of the two at (-0.080 against +0.044). A small true
lets the extended model fit sizeable inefficiency of either sign
to noise. It is the better choice when the wrong skewness is the
problem, not a free improvement everywhere.
esfm() returns an object of class "esfm" (and "sfareg").
The components most often needed:
out |
matrix of par, st_err, t-val for
gamma, sigv and the frontier coefficients |
gamma, sigma_v |
the two scale estimates; gamma < 0
means a wrongly skewed fit |
Efficiency |
for each firm |
logLik, logLik_restricted |
log-likelihood, and that of the
(OLS) fit |
lr_symmetry, p_symmetry |
the LR statistic for
and its p-value |
wrong_skew_sample, residual_m3 |
whether the OLS residuals are wrongly skewed, and their third moment |
model_name |
"EXT-HALFNORMAL" or "EXT-EXPONENTIAL"
|
symmetry_test() returns a one-row data frame with columns test,
statistic, df (always 1), null ("chi2(1)") and
p.value, read from the fit's lr_symmetry and p_symmetry.
Hafner, C. M., Manner, H. and Simar, L. (2018). The “wrong skewness” problem in stochastic frontier models: A new approach. Econometric Reviews 37(4), 380–400.
Almanidis, P., Qian, J. and Sickles, R. C. (2014). Stochastic frontier models with bounded inefficiency. In Festschrift in Honor of Peter Schmidt.
skewness_test, inefficiency_test, sfm
set.seed(3) n <- 200 lx1 <- rnorm(n, 1.5, 0.3); lx2 <- rnorm(n, 1.8, 0.3) y <- 0.9 + 0.6 * lx1 + 0.5 * lx2 + rnorm(n, 0, 0.25) - abs(rnorm(n, 0, 0.5)) d <- data.frame(y, lx1, lx2) f <- esfm(y ~ lx1 + lx2, data = d) f symmetry_test(f)set.seed(3) n <- 200 lx1 <- rnorm(n, 1.5, 0.3); lx2 <- rnorm(n, 1.8, 0.3) y <- 0.9 + 0.6 * lx1 + 0.5 * lx2 + rnorm(n, 0, 0.25) - abs(rnorm(n, 0, 0.5)) d <- data.frame(y, lx1, lx2) f <- esfm(y ~ lx1 + lx2, data = d) f symmetry_test(f)
Cross-sectional data on Finnish electricity distribution firms, including annual averages of expenditure and output measures over a four-year regulatory period.
data("FinnishElec")data("FinnishElec")
A data frame with 89 observations on the following 6 variables.
ida character vector containing a unique identifier for each distribution firm
xa numeric vector containing total expenditure (TOTEX*) (1000 Euros)
y1a numeric vector containing weighted energy transmitted through the network (GWh of 0.4 kV equivalents)
y2a numeric vector containing total length of the network (km)
y3a numeric vector containing total number of customers connected to the network
za numeric vector containing the proportion of underground cables in the total network length.
*TOTEX includes capital expenditure (CAPEX), controllable operational expenditure (OPEX), and estimated external cost of interruptions.
Kuosmanen, T. (2012). 'Stochastic semi-nonparametric frontier estimation of electricity distribution networks: Application of the StoNED method in the Finnish regulatory model.' Energy Economics, 34(6), pp. 2189-2199. doi:10.1016/j.eneco.2012.03.005
data(FinnishElec) plot(FinnishElec)data(FinnishElec) plot(FinnishElec)
Tests the distributional assumption on , holding the normality of
fixed, following Wang, Amsler and Schmidt (2011).
gof_test(object, data = NULL, test = c("ks", "chisq"), null = c("bootstrap", "asymptotic"), B = 199, cells = 10, seed = NULL)gof_test(object, data = NULL, test = c("ks", "chisq"), null = c("bootstrap", "asymptotic"), B = 199, cells = 10, seed = NULL)
object |
An |
data |
The data the model was fitted to. |
test |
|
null |
|
B |
Bootstrap replications. Each one refits the model. |
cells |
Number of equiprobable cells for the chi-square test. |
seed |
Optional seed for the bootstrap. |
The distribution of is the assumption in this model least often
defended and least often tested, and it is testable: with the noise
distribution maintained, the assumed implies a distribution for the
composed error, so rejecting that is rejecting the assumed .
The test is on , not on . Wang, Amsler
and Schmidt point out that is a
monotonic function of , so the KS test is identical either
way and the chi-square test is identical when the cells are defined
conformably – but is much easier to work with. It is a
mistake, not a diagnostic, to compare the observed spread of
with the assumed density of : those are different distributions.
Parameter estimation cannot be ignored. Both statistics are evaluated
at , which changes their null distributions. The default
"bootstrap" handles this by copying the estimation step exactly: each
replication draws a composed error from the fitted model, rebuilds the
response, refits, forms its own residuals, and recomputes the
statistic. "asymptotic" returns a p-value for the
chi-square statistic, which the authors note is conservative when
evaluated at the MLE – that reference distribution belongs to the
minimum-chi-square estimator. There is no asymptotic KS p-value here: with
estimated the Kolmogorov distribution does not apply, and Bai's
(2003) martingale transformation is not implemented, so NA is returned
rather than a number that would look valid.
Size and power. Measured in this package over 200 replications with
B = 49, half-normal data fitted as half-normal for size and
exponential data fitted as half-normal for power:
|
KS .10 | KS .05 | chi2 .10 | chi2 .05 | |
| size | 200 | 0.065 | 0.025 | 0.085 | 0.065 |
| size | 800 | 0.070 | 0.040 | 0.065 | 0.010 |
| power | 200 | 0.505 | 0.395 | 0.235 | 0.145 |
| power | 800 | 0.985 | 0.965 | 0.810 | 0.655 |
| power | 2000 | 1.000 | 1.000 | 0.990 | 0.985 |
Both tests hold their size, mildly conservatively. The KS test
dominates the chi-square test at every sample size, which is the paper's own
conclusion and the reason it is the better default: the chi-square statistic
discards the within-cell information that KS uses. Power against the
half-normal/exponential pair – one of the harder distinctions, since both
are one-parameter and similarly shaped – is about 0.4 at and
essentially 1 by , so a failure to reject in a small sample is
weak evidence.
A data frame with one row per test: test, statistic,
null, p.value and B (replications actually used).
Wang, W. S., Amsler, C. and Schmidt, P. (2011). Goodness of fit tests in stochastic frontier models. Journal of Productivity Analysis 35, 95–118.
Bai, J. (2003). Testing parametric conditional distributions of dynamic models. Review of Economics and Statistics 85, 531–549.
spec_test, pcomposed_model, sfm
d <- as.data.frame(data_gen_cs(N = 200, rand = 5, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1)) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) ## Every bootstrap replication REFITS, so keep B small in an example; use ## B = 999 for a real test. gof_test(f, data = d, B = 39, seed = 1) gof_test(f, data = d, test = "chisq", null = "asymptotic")d <- as.data.frame(data_gen_cs(N = 200, rand = 5, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1)) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) ## Every bootstrap replication REFITS, so keep B small in an example; use ## B = 999 for a real test. gof_test(f, data = d, B = 39, seed = 1) gof_test(f, data = d, test = "chisq", null = "asymptotic")
Evaluates the Hyvarinen score used to choose the robustness tuning parameter of
the maximum -likelihood and minimum density-power divergence
estimators, following Sugasawa and Yonekura (2021). Lower is better.
hscore(e, sigma_v, sigma_u, c, stable = TRUE)hscore(e, sigma_v, sigma_u, c, stable = TRUE)
e |
Numeric vector of composed residuals |
sigma_v, sigma_u
|
Positive scalars, the noise and inefficiency scale parameters. |
c |
Robustness tuning parameter, |
stable |
Logical. |
The score is
with derivatives taken with respect to the composed residual.
The score is evaluated in logarithms. The natural-scale expression contains the
fitted density raised to the power ; since that exponent is
close to , so a single observation whose density underflows makes the
score non-finite. The failure is silent and directional: the tuning candidates
that remain evaluable are those closest to maximum likelihood, so a naive
implementation reports too little robustness. Set stable = FALSE to
reproduce that behaviour for comparison.
A single numeric value, the mean observationwise score.
Hyvarinen, A. (2005). Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research 6, 695–709.
Sugasawa, S. and Yonekura, S. (2021). On selection criteria for the tuning parameter in robust divergence. Entropy 23(9), 1147.
Bernstein, D.H., Parmeter, C.F. and Wright, I.A. (2026). On Robust Estimation of the Stochastic Frontier Model. Working paper.
hscore_select, calibrate_c,
density_weights
set.seed(1) e <- rnorm(500, 0, 0.3) - abs(rnorm(500, 0, 0.6)) hscore(e, sigma_v = 0.3, sigma_u = 0.6, c = 0.20) ## the two evaluations agree wherever both are computable hscore(e, 0.3, 0.6, 0.20) - hscore(e, 0.3, 0.6, 0.20, stable = FALSE) ## but only one of them survives an observation in the far tail e2 <- c(e, -60) hscore(e2, 0.3, 0.6, 0.15) # finite hscore(e2, 0.3, 0.6, 0.15, stable = FALSE) # not finiteset.seed(1) e <- rnorm(500, 0, 0.3) - abs(rnorm(500, 0, 0.6)) hscore(e, sigma_v = 0.3, sigma_u = 0.6, c = 0.20) ## the two evaluations agree wherever both are computable hscore(e, 0.3, 0.6, 0.20) - hscore(e, 0.3, 0.6, 0.20, stable = FALSE) ## but only one of them survives an observation in the far tail e2 <- c(e, -60) hscore(e2, 0.3, 0.6, 0.15) # finite hscore(e2, 0.3, 0.6, 0.15, stable = FALSE) # not finite
Refits a Normal–Half-Normal stochastic frontier at each candidate value of the
robustness tuning parameter and returns the value minimising the Hyvarinen score
of hscore. The search is coarse-to-fine and includes the
maximum-likelihood endpoint , so the criterion is free to decline
robustification altogether.
hscore_select(object, method = c("mlqe", "psi", "mdpd"), range = c(0, 0.60), coarse = 0.01, fine = 0.001, half_width = 0.02, multistart = TRUE, verbose = FALSE)hscore_select(object, method = c("mlqe", "psi", "mdpd"), range = c(0, 0.60), coarse = 0.01, fine = 0.001, half_width = 0.02, multistart = TRUE, verbose = FALSE)
object |
A fitted model from |
method |
Robust criterion. |
range |
Numeric length-2 search interval. |
coarse, fine
|
Grid steps for the two passes. |
half_width |
Half-width of the refinement window around the coarse minimum. |
multistart |
Logical; see Details. |
verbose |
Logical, print progress. |
Two failure modes are guarded here, both of which return a converged fit and a finite criterion value if they are not.
Silent grid loss. The criterion is evaluated in log space (see
hscore). Evaluated naively it is non-finite wherever any fitted
density underflows, which discards candidates non-randomly – those nearest
maximum likelihood survive – and so understates the robustness the data call
for.
Warm-start capture. Walking the grid upward and warm-starting each fit
from the previous one is efficient but unguarded: if one fit falls into the
degenerate basin, every later fit inherits it. With
multistart = TRUE each candidate is fitted from both the warm start and a
fixed reference start, keeping the better objective.
An object of class "sfa_hscore": the selected tuning parameter c,
its criterion value hscore, the whole search path as a data frame,
the refitted model fit, and n_finite, how many candidates were
scorable out of how many tried.
A flat region around the minimum is not non-identification in the usual sense; it
means the criterion discriminates weakly among nearby tuning values. The drop
from to the minimum is a descriptive score difference, not a test of
maximum likelihood against a robust alternative. Where the selected value
matters, compare it with the fixed calibration from calibrate_c and
check its stability by resampling: the selection can be bimodal across
resamples even when the full-sample criterion clearly prefers a robust solution.
Sugasawa, S. and Yonekura, S. (2021). On selection criteria for the tuning parameter in robust divergence. Entropy 23(9), 1147.
Bernstein, D.H., Parmeter, C.F. and Wright, I.A. (2026). On Robust Estimation of the Stochastic Frontier Model. Working paper.
hscore, calibrate_c,
density_weights, sfm
set.seed(3) n <- 200 x <- runif(n, 1, 10) y <- 1 + 0.5 * log(x) + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.6)) dat <- data.frame(y = y, x = log(x)) fit <- sfm(y ~ x, data = dat, model_name = "NHN") sel <- hscore_select(fit, method = "mlqe") sel plot(sel$path$c, sel$path$hscore, type = "l", xlab = "c", ylab = "Hyvarinen score") abline(v = sel$c, lty = 3)set.seed(3) n <- 200 x <- runif(n, 1, 10) y <- 1 + 0.5 * log(x) + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.6)) dat <- data.frame(y = y, x = log(x)) fit <- sfm(y ~ x, data = dat, model_name = "NHN") sel <- hscore_select(fit, method = "mlqe") sel plot(sel$path$c, sel$path$hscore, type = "l", xlab = "c", ylab = "Hyvarinen score") abline(v = sel$c, lty = 3)
Panel data on 34 paddy farmers from Aurepalle, India, collected over ten years (1975-76 to 1984-85). The panel is unbalanced: farmers are observed between 2 and 10 times. Includes farmer characteristics (age, schooling) and production variables (output, land, labor, bullocks, input costs).
data("Indian")data("Indian")
A data frame with 273 observations (an unbalanced panel of 34 farmers over 10 years) on the following 10 variables.
ida numeric vector containing a unique identifier for each farmer
yra numeric vector containing the year of the observation
agea numeric vector containing the age of the primary decision maker
schoola numeric vector containing the number of years of schooling of the primary decision maker
yvara numeric vector containing the natural logarithm of the total value of output (rupees)
Llanda numeric vector containing the natural logarithm of the total area of land operated (ha)
PIlanda numeric vector containing the proportion of land that is irrigated
Llabora numeric vector containing the natural logarithm of the total number of hours of hired and family labour used
Lbulla numeric vector containing the natural logarithm of the number of hours of bullock labour used
Lcosta numeric vector containing the natural logarithm of the value of inputs including fertilizer, manure, pesticides, machinery, etc.
Battese, G.E. and Coelli, T.J. (1995) 'A model for technical inefficiency effects in a stochastic frontier production function for panel data', Empirical Economics, 20(2), pp. 325-332. doi:10.1007/BF01205442.
Battese, G.E. and Coelli, T.J. (1992) 'Frontier production functions, technical efficiency and panel data: With application to paddy farmers in India', Journal of Productivity Analysis, 3(1-2), pp. 153-169. doi:10.1007/BF00158774.
data(Indian)data(Indian)
Tests : no technical inefficiency, i.e. in the
Battese-Corra parameterization , following Section 3 of Coelli (1995).
inefficiency_test(object, test = c("lr_1sided", "lr", "wald", "m3t"), level = 0.05)inefficiency_test(object, test = c("lr_1sided", "lr", "wald", "m3t"), level = 0.05)
object |
An |
test |
Which statistics to compute; any subset, all by default. |
level |
Significance level used for the |
The two tests most often reported are the two with the wrong size.
puts on the boundary of the parameter space, so the
usual asymptotics fail. The likelihood ratio statistic is not
but the Gourieroux, Holly and Monfort (1982) mixture
, whose 5% critical value is
2.71 rather than 3.84; using the value makes the naive LR
test conservative. The Wald ratio is worse:
it is not asymptotically standard normal at the boundary at all.
Measured in this package on 400 replications per cell, half-normal data with
and a nominal 5% level:
|
LR one-sided | LR naive | Wald | M3T |
| 100 | 0.060 | 0.030 | 0.230 | 0.055 |
| 200 | 0.060 | 0.033 | 0.203 | 0.050 |
| 400 | 0.055 | 0.033 | 0.217 | 0.045 |
| 800 | 0.060 | 0.035 | 0.182 | 0.062 |
The one-sided LR test and the third-moment test hold their size; the naive
LR test rejects about half as often as it should; the Wald test rejects
three to four times too often and does not improve with . Under the
alternative () the one-sided LR test has the
better power of the two valid tests at every sample size, which is Coelli's
recommendation. The Wald column's apparently high power is not
comparable, because its size is wrong.
For model_name = "NTN" the null restricts both and
, and the mixture becomes
.
The "m3t" statistic is the same one skewness_test
reports as test = "coelli"; it is repeated here so the four tests can
be compared in a single table.
A data frame with one row per test: test, statistic,
null (the reference distribution), p.value and reject.
The log-likelihoods under both hypotheses are attached as attributes.
Coelli, T. (1995). Estimators and hypothesis tests for a stochastic frontier function: A Monte Carlo analysis. Journal of Productivity Analysis 6, 247–268.
Gourieroux, C., Holly, A. and Monfort, A. (1982). Likelihood ratio test, Wald test, and Kuhn-Tucker test in linear models with inequality constraints on the regression parameters. Econometrica 50, 63–80.
d <- data_gen_cs(N = 300, rand = 7, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = as.data.frame(d)) inefficiency_test(f)d <- data_gen_cs(N = 300, rand = 7, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = as.data.frame(d)) inefficiency_test(f)
Empirical influence function and case-influence measures for a fitted stochastic frontier. Answers which observations move the estimates, and whether the specification lets any single observation move them without bound.
influence_sfa(object, scale = TRUE)influence_sfa(object, scale = TRUE)
object |
An |
scale |
Multiply the influence function by |
The standard outlier rules do not transfer to this model. Least-median-of-squares
and its relatives ignore the asymmetry of the composed error, so a
genuinely large – an inefficient firm, the thing being measured –
reads as an outlier and is discarded, which defeats the analysis.
The influence function is the right object instead. Maximum likelihood is an
M-estimator, so its influence function is a linear transformation of the score,
, and the estimator is
B-robust exactly when that is bounded. Both pieces are already available:
estfun supplies the per-observation scores and
vcov the inverse information.
Which sensitivity to read. The raw sup-norm depends on how the parameters
happen to be scaled, so it is not comparable between models with different
parameter vectors. On a clean sample of 200 it reads 72.6 for "NHN" and
1322.5 for "tHN", which appears to say the robust specification is far
worse; all it reflects is tHN's extra on its own scale.
sensitivity_std measures the same influence function in the information
metric and is invariant to reparameterisation. On that scale, contaminating a
single response takes "NHN" from 13.5 to 32.3 and "tHN" from 13.1
to 17.2 – Stead, Wheat and Greene's finding, and invisible in the raw number.
Neither number has an absolute threshold. The comparison to make is between specifications on the same data, not either number against a cut-off.
What to do about a large value. Nothing, by deletion. A large sensitivity
is a property of the specification: under an unbounded influence function the
next most extreme point simply replaces the one removed. The remedies are a
specification with bounded influence – Stead, Wheat and Greene's Student's
noise term, model_name = "tHN" – or a robust divergence
criterion, sfm(robust = ), whose tuning is chosen by
hscore_select or calibrate_c.
What this cannot see. Like density_weights, it is computed
at the fitted surface. An observation mis-recorded in a regressor can bend
that surface toward itself and so appear unremarkable here. Pair this with a
leverage check before concluding that an observation is not driving a result.
An object of class "sfa_influence":
influence |
|
d |
Length- |
sensitivity_std |
|
sensitivity |
|
max_abs_by_parameter |
Largest |
Stead, A.D., Wheat, P. and Greene, W.H. (2023). Robust maximum likelihood estimation of stochastic frontier models. European Journal of Operational Research, 309(1), 188–201.
Bernstein, D.H., Parmeter, C.F. and Wright, I.A. (2026). On Robust Estimation of the Stochastic Frontier Model. Working paper.
density_weights, hscore_select,
sfm, sfareg-methods
set.seed(3) n <- 120 x <- runif(n, 1, 10) d <- data.frame(y = 1 + 0.5 * log(x) + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.6)), x = log(x)) fit <- sfm(y ~ x, data = d, model_name = "NHN", keep_objective = TRUE) influence_sfa(fit) ## Contaminate one response and compare specifications on the ## self-standardised scale. d2 <- d; d2$y[1] <- d2$y[1] - 5 a <- sfm(y ~ x, data = d2, model_name = "NHN", keep_objective = TRUE) b <- sfm(y ~ x, data = d2, model_name = "tHN", keep_objective = TRUE) c(NHN = influence_sfa(a)$sensitivity_std, tHN = influence_sfa(b)$sensitivity_std)set.seed(3) n <- 120 x <- runif(n, 1, 10) d <- data.frame(y = 1 + 0.5 * log(x) + rnorm(n, 0, 0.3) - abs(rnorm(n, 0, 0.6)), x = log(x)) fit <- sfm(y ~ x, data = d, model_name = "NHN", keep_objective = TRUE) influence_sfa(fit) ## Contaminate one response and compare specifications on the ## self-standardised scale. d2 <- d; d2$y[1] <- d2$y[1] - 5 a <- sfm(y ~ x, data = d2, model_name = "NHN", keep_objective = TRUE) b <- sfm(y ~ x, data = d2, model_name = "tHN", keep_objective = TRUE) c(NHN = influence_sfa(a)$sensitivity_std, tHN = influence_sfa(b)$sensitivity_std)
Fits the stochastic frontier model of Amsler, Prokhorov and Schmidt (2016) in
which one or more regressors is endogenous: correlated with the noise
, not with the inefficiency. With uhet the model becomes that of
Amsler, Prokhorov and Schmidt (2017), in which environmental variables
entering the inefficiency scale may themselves be endogenous. Three
estimators are provided:
model_name |
estimator | when to use it |
"IVLIML" (default) |
full-information maximum likelihood | efficient, and its standard errors are correct as reported |
"IVCF" |
two-step control function (Kutlu, 2010) | cheaper; its standard errors are too small when the endogeneity is real |
"C2SLS" |
corrected 2SLS | closed form, no likelihood; no uhet
|
ivsfm(formula, endogenous, instruments, data, model_name = c("IVLIML", "IVCF", "C2SLS"), uhet = NULL, inefdec = TRUE, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)ivsfm(formula, endogenous, instruments, data, model_name = c("IVLIML", "IVCF", "C2SLS"), uhet = NULL, inefdec = TRUE, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)
formula |
The structural frontier, |
endogenous |
One-sided formula naming the endogenous variables,
|
instruments |
One-sided formula of the excluded instruments,
|
data |
A |
model_name |
|
uhet |
Optional one-sided formula of environmental variables, giving
|
inefdec |
|
maxit.bobyqa, maxit.psoptim, maxit.optim
|
Iteration caps for the three
optimizer stages. Ignored by |
start_val |
Name the starting-value vector in the returned object. |
PSopt |
Run the particle-swarm stage between BOBYQA and |
optHessian |
Compute the Hessian, and with it the standard errors. |
Method |
Method passed to |
verbose |
Report optimizer progress. |
keep_objective |
Logical. Retain the likelihood closure on the fitted object as |
rand.psoptim |
Optional seed for the particle-swarm stage. |
Unlike the rest of the package neither formula nor endogenous nor
instruments takes a | segment. A pipe means “variance
determinant” in sfm and psfm, and reusing it here
for ivreg-style instrument syntax would give the same character two
meanings across entry points, so a pipe is an error rather than being
reinterpreted.
Writing for the endogenous block and for
the instruments,
with independent of . Endogeneity is the
correlation between and : the regressors are correlated with
the noise, not with the inefficiency.
Conditioning on turns this into an ordinary normal–half-normal
frontier in a shifted residual with a smaller noise variance,
which is what makes the likelihood closed-form and needs no simulation.
"IVLIML"APS (2016) Equation (13): maximum likelihood over the
frontier and the reduced form jointly, so and
are estimated alongside . Efficient, and
its standard errors are correct as reported. It is seeded from the
"IVCF" solution rather than from , which is why fitting
it runs a cheap control-function fit first. is
exactly – the point where the frontier and reduced-form
blocks are independent, so the likelihood has no gradient in
there – and it is also the centre of the search box, which put the corner
exactly as far from the start as the true
optimum. Through version 1.2.0 the optimizer took that corner on a minority
of samples and reported the result as an ordinary fit. Since "IVCF"
is this same likelihood with and held
at least squares, its answer is a point of "IVLIML"'s own parameter
space, not an approximation carried in from elsewhere.
"IVCF"The two-step control function of Kutlu (2010), described
in APS (2016) Section 4.4. and are fixed at
their reduced-form least-squares values and only the frontier block is
maximized. Cheaper, still consistent, but it discards the information about
the reduced form carried in the frontier likelihood; its conventional
standard errors understate the true ones unless .
Measured over 30 replications at : with
the mean reported standard error on the endogenous coefficient is 0.912 of
the Monte Carlo standard deviation, while "IVLIML"'s is 1.005; with
the two are identical, as the theory says they should be.
Bootstrap, or use "IVLIML", when the standard errors matter.
"C2SLS"APS (2016) Section 4.1: 2SLS, then and
from the second and third moments of the residuals, then the
intercept corrected by . The residuals use
the actual endogenous regressors, not their fitted values, which the
paper flags explicitly. Moment-based rather than likelihood-based, so the
fit carries no $opt and logLik returns NA with
a warning – the same footing as psfm's "GTRE_SEQ1". It
estimates no , since it never models the correlation it corrects
for.
jlms implements APS (2016) Equations (14)–(15),
which condition on as well as on . Since
is correlated with it carries information about even though
is independent of , so this is a strictly better predictor
than the plain Jondrow et al. (1982) formula.
is assumed independent
of . The case where the reduced-form error is correlated with the
inefficiency as well requires the copula likelihood of APS (2016) Section 4.5
and is not implemented; the paper notes there that two intuitive
approaches to that case do not work.
Standard errors for arrived in 1.2.0. They were previously
reported as NA, on the ground that has a
non-diagonal Jacobian and a delta-method value computed as though it were
diagonal would be wrong. That much was true, but the Jacobian itself is short –
with – so
is available
exactly. Checked against a numerical Jacobian in the tests, and validated
end-to-end by the size of endogeneity_test, which is 0.046 at a
nominal 5% over 1000 replications.
An object of class "sfareg". out is a p x 3 matrix, one
row per parameter, columns par, st_err and t-val.
For "IVLIML" and "IVCF" the parameters are the frontier
coefficients, sigma_u, sigma_v, any delta_ terms from
uhet, and one rho_ per endogenous variable. For "C2SLS"
they are sigma_u and sigma_v first, then the frontier
coefficients, all with NA standard errors; it reports no rho_,
and the fit carries no rho, vcov_rho or sigma_c.
Alongside the usual components:
Pi |
Reduced-form coefficient matrix: estimated jointly for
|
Sigma_xi |
Reduced-form error covariance, on the same footing as
|
rho |
Correlation of each reduced-form error with |
sigma_c |
The conditional noise standard deviation |
b_2sls |
The 2SLS coefficients, which seed the likelihood and are the
estimate |
wrong_skew |
Whether the 2SLS residuals are skewed the wrong way. |
jlms, efficiency
|
|
vcov_rho |
The full |
Amsler, C., Prokhorov, A. and Schmidt, P. (2016). Endogeneity in stochastic frontier models. Journal of Econometrics, 190(2), 280–288.
Amsler, C., Prokhorov, A. and Schmidt, P. (2017). Endogenous environmental variables in stochastic frontier models. Journal of Econometrics, 199(2), 131–140.
Kutlu, L. (2010). Battese-Coelli estimator with endogenous regressors. Economics Letters, 109(2), 79–81.
endogeneity_test for the Wald test of whether the
correction was needed at all; sfm for the frontier without one.
set.seed(1) n <- 600 w1 <- rnorm(n); w2 <- rnorm(n); x1 <- rnorm(n) eta <- rnorm(n) v <- 0.5 * (0.6 * eta + sqrt(1 - 0.6^2) * rnorm(n)) x2 <- 0.9 * w1 - 0.7 * w2 + 0.5 * x1 + eta y <- 0.5 + 0.8 * x1 - 0.6 * x2 + v - abs(rnorm(n)) dat <- data.frame(y = y, x1 = x1, x2 = x2, w1 = w1, w2 = w2) fit <- ivsfm(y ~ x1 + x2, endogenous = ~x2, instruments = ~ w1 + w2, data = dat) fit$out ## Ignoring the endogeneity biases the coefficient on x2 toward zero. sfm(y ~ x1 + x2, model_name = "NHN", data = dat)$outset.seed(1) n <- 600 w1 <- rnorm(n); w2 <- rnorm(n); x1 <- rnorm(n) eta <- rnorm(n) v <- 0.5 * (0.6 * eta + sqrt(1 - 0.6^2) * rnorm(n)) x2 <- 0.9 * w1 - 0.7 * w2 + 0.5 * x1 + eta y <- 0.5 + 0.8 * x1 - 0.6 * x2 + v - abs(rnorm(n)) dat <- data.frame(y = y, x1 = x1, x2 = x2, w1 = w1, w2 = w2) fit <- ivsfm(y ~ x1 + x2, endogenous = ~x2, instruments = ~ w1 + w2, data = dat) fit$out ## Ignoring the endogeneity biases the coefficient on x2 toward zero. sfm(y ~ x1 + x2, model_name = "NHN", data = dat)$out
Fits finite-mixture stochastic frontiers: models in which each firm belongs to one of unobserved classes, and the likelihood mixes the classes' normal/half-normal densities weighted by the class probabilities. Three specifications:
"LCM": the latent class frontier of Greene (2005) and Orea and Kumbhakar (2004). Each class has its own frontier and its own and ; the class probabilities are constant.
"LCM_Z": "LCM" with the class probabilities depending on covariates, given in the second part of the formula.
"LCM_CN": the contaminated normal frontier. One frontier and one ; only the noise scale differs between components, which gives the noise heavier tails.
lcsfm(formula, model_name = c("LCM", "LCM_Z", "LCM_CN"), data, n_class = 2, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, penalty_c = 0, upper = NA, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)lcsfm(formula, model_name = c("LCM", "LCM_Z", "LCM_CN"), data, n_class = 2, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, penalty_c = 0, upper = NA, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)
formula |
A symbolic description of the frontier, |
model_name |
|
data |
A data frame containing the variables named in |
n_class |
Integer. The number of latent classes |
maxit.bobyqa |
Maximum number of iterations for the bobyqa optimization routine. |
maxit.psoptim |
Maximum number of iterations for the psoptim optimization routine. |
maxit.optim |
Maximum number of iterations for the optim optimization routine. |
REPORT, trace, pgtol, upper
|
Accepted for compatibility with the other entry points' signatures, but not used by |
start_val |
|
PSopt |
Logical. Run the particle-swarm stage ( |
optHessian |
Logical. Finish with |
inefdec |
Logical. |
penalty_c |
Tuning constant for Chen et al.'s (2001) penalty |
Method |
Method passed to |
verbose |
Logical. Print optimization progress messages? Default is |
keep_objective |
Logical. Retain the likelihood closure on the fitted object as |
rand.psoptim |
Integer. Seed for replication of psoptim. Defaults to |
"LCM", "LCM_Z") distinct technologies are present in one sample, and which one a firm operates is unobserved. Each class has its own frontier and its own two scales,
and class membership follows a multinomial logit,
with class the reference. Under "LCM" the are a constant, so every firm has the same class probabilities; under "LCM_Z" they are the covariates in the second part of the formula. Every firm contributes to every class, weighted by its class probability, so the log-likelihood mixes the composed densities rather than assigning firms to groups:
This is not the same as fitting separate frontiers to known subgroups: the groups are unknown, and the uncertainty about them is carried into the efficiency predictions. It is also more general than zsfm's zero-inefficiency model, which is the restricted two-class case in which one class has no inefficiency at all.
post.prob is an matrix of posterior class probabilities. jlms_class holds the class-conditional Jondrow et al. (1982) predictions, and jlms is their posterior-weighted average. A firm's inefficiency is only defined relative to a frontier, and which frontier it faces is itself uncertain, so reporting the modal class alone would discard that. class gives the modal class, and class_prob the average prior class shares.
"LCM_CN")The same machinery, used to give the noise heavier tails rather than to separate technologies. The noise density is a scale mixture of normals, , while the frontier, and the mixing proportions are common to all components. At this is the scale-contaminated normal, an alternative to a Student- noise term.
It has a closed form, which is why it is cheap. Since is linear in , a mixture noise density passes straight through:
a mixture of ordinary normal/half-normal densities sharing one . No extra integration is required, and the tests check the identity against direct numerical integration.
The parameter vector is rather than one block per class. There is one frontier and one , so there are no class-conditional efficiency predictions: jlms, jlms_class and class are not returned.
This is also the specification lcsfm_homogeneity's asymptotic null is stated for: one scalar parameter differing between components. On "LCM", where everything varies, the null rejected 63.5% of the time at a nominal 5% under a true null; on "LCM_CN" it rejected 8.0%. That is still mildly liberal, and the test says so, but usable.
Class labels in a finite mixture are identified only up to permutation: relabelling the classes and permuting gives the identical likelihood. Starting values split the OLS residuals at their -quantiles and refit within each group. That makes a run reproducible on the same data, and it avoids the saddle point that identical starting components would sit at (equal posterior class probabilities give a zero score with respect to the class split). It does not pin the labels to any external ordering. Do not compare class 1 across two fits, or against a simulation's true labels, without matching the classes first.
The likelihood cannot fall when a class is added, and the usual regularity conditions for a likelihood-ratio test of against do not hold, since the null puts a class probability on the boundary. Use AIC()/BIC(), the separation of the fitted frontiers, and whether the extra class is economically interpretable, rather than a p-value. Whether any latent classes are needed at all – one class against the fitted – is the exception: lcsfm_homogeneity tests it for "LCM" and "LCM_CN" fits.
An object of class "sfareg" containing the following components:
out |
A matrix with parameter estimates, standard errors, and t-values, one row per parameter. Under |
opt |
A list containing the optimization results from the final optimization procedure. With |
total_time |
The total computation time for model estimation. |
start_v |
The final parameter vector, in the order of the rows of |
model_name |
The specification fitted: |
formula |
The formula used in the model specification. |
jlms, jlms_class, class
|
|
post.prob |
An |
class_prob |
Average prior class probabilities, one per class. |
n_class |
The number of classes fitted. |
penalty, logLik_unpenalised, penalty_c
|
The penalty term at the estimate, the log-likelihood without it, and the |
coefficients |
A vector of estimated parameters. |
std.errors |
A vector of standard errors for the estimated parameters (NA if |
t.values |
A vector of t-values for the estimated parameters (NA if |
call |
The matched call. |
nobs |
The number of observations used in the fit. |
objective |
The likelihood closure, when |
Standard errors require optHessian = TRUE, the default.
Christopher F. Parmeter and David H. Bernstein
Greene, W. (2005) 'Reconsidering heterogeneity in panel data estimators of the stochastic frontier model', Journal of Econometrics, 126(2), pp. 269-303. doi:10.1016/j.jeconom.2004.05.003.
Orea, L. and Kumbhakar, S.C. (2004) 'Efficiency measurement using a latent class stochastic frontier model', Empirical Economics, 29(1), pp. 169-183. doi:10.1007/s00181-003-0184-2.
Caudill, S.B. (2003) 'Estimating a mixture of stochastic frontier regression models via the em algorithm: a multiproduct cost function application', Empirical Economics, 28(3), pp. 581-598. doi:10.1007/s001810200142.
Chen, H., Chen, J. and Kalbfleisch, J.D. (2001) 'A modified likelihood ratio test for homogeneity in finite mixture models', Journal of the Royal Statistical Society: Series B, 63(1), pp. 19-29.
Jondrow, J., Lovell, C.A.K., Materov, I.S. and Schmidt, P. (1982) 'On the estimation of technical inefficiency in the stochastic frontier production function model', Journal of Econometrics, 19(2-3), pp. 233-238. doi:10.1016/0304-4076(82)90004-5.
lcsfm_homogeneity, zsfm, sfm, psfm, ttsfm, sfareg-methods
library(sfa) ## Two coexisting technologies, class probabilities constant. set.seed(42) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) cls <- rbinom(n, 1, 0.4) + 1L b <- rbind(c(1, 0.5, 0.5), c(3, 1.0, 0.2)) sv <- c(0.2, 0.2); su <- c(0.5, 1.0) y <- b[cls, 1] + b[cls, 2] * x1 + b[cls, 3] * x2 + rnorm(n, 0, sv[cls]) - abs(rnorm(n, 0, su[cls])) fit <- lcsfm(y ~ x1 + x2, model_name = "LCM", data = data.frame(y, x1, x2), n_class = 2) fit$class_prob # average class shares head(fit$post.prob) # posterior class membership, one row per firmlibrary(sfa) ## Two coexisting technologies, class probabilities constant. set.seed(42) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) cls <- rbinom(n, 1, 0.4) + 1L b <- rbind(c(1, 0.5, 0.5), c(3, 1.0, 0.2)) sv <- c(0.2, 0.2); su <- c(0.5, 1.0) y <- b[cls, 1] + b[cls, 2] * x1 + b[cls, 3] * x2 + rnorm(n, 0, sv[cls]) - abs(rnorm(n, 0, su[cls])) fit <- lcsfm(y ~ x1 + x2, model_name = "LCM", data = data.frame(y, x1, x2), n_class = 2) fit$class_prob # average class shares head(fit$post.prob) # posterior class membership, one row per firm
Tests whether a lcsfm fit's latent classes are needed at all, against the null that one class describes the data. This is the first question a latent class fit raises and the ordinary likelihood ratio test cannot answer it, because under the null the second class's parameters are unidentified, the mixing proportion sits on a boundary, and the classes are interchangeable.
lcsfm_homogeneity(object, null = c("bootstrap", "chisq01"), B = 199, c = 1, level = 0.05, seed = NULL, quiet = FALSE, envir = parent.frame())lcsfm_homogeneity(object, null = c("bootstrap", "chisq01"), B = 199, c = 1, level = 0.05, seed = NULL, quiet = FALSE, envir = parent.frame())
object |
An |
null |
How to obtain the null distribution. |
B |
Bootstrap replications. Each is two refits, so this dominates the running time. The smallest attainable p-value is |
c |
Tuning constant in the penalty |
level |
Size used for the reported verdict. |
seed |
Optional integer seed for the bootstrap. The RNG state is saved and restored, so a seeded call does not disturb the caller's stream. |
quiet |
Set |
envir |
Environment in which to re-evaluate the fit's |
The statistic. lcsfm is refitted maximising the modified likelihood – the ordinary log-likelihood plus Chen et al.'s penalty , which diverges to as any class probability approaches 0 or 1 and so keeps every class occupied. The statistic is twice the difference between that maximum and the maximised log-likelihood of the one-class model, which is sfm(model_name = "NHN") on the same formula, since a single latent class is the ordinary frontier. The penalty is exactly zero at equal class probabilities, so the null model is recovered without a handicap.
Why the default null is a bootstrap. Stead, Wheat and Greene report the modified statistic is asymptotically , a 50:50 mixture of a point mass at zero and . That result is stated for the case in which a scalar parameter differs between classes – Zhu and Zhang (2004) require and scalar, and the paper's own application holds every parameter except the noise scale common across classes.
lcsfm's "LCM" does not do that: it lets , and the whole frontier vector vary by class. The limit therefore does not carry over, and this was measured rather than assumed. Over 200 replications of a one-class data generating process at :
| nominal size | 10% | 5% |
actual rejection under "chisq01" |
82.0% | 63.5% |
with the sampling distribution tracking – exactly the parameter-count difference – rather than , whose median is 0 against an observed 4.21.
A parametric bootstrap makes no appeal to an asymptotic distribution: it simulates from the fitted null and re-runs the whole procedure, so it is valid whatever the parameter structure. Over 60 replications of the same one-class process with B = 99:
| nominal size | 10% | 5% |
"chisq01" |
82.0% | 63.5% |
"bootstrap" |
10.0% | 6.7% |
both within a binomial standard error of nominal, with the p-values indistinguishable from uniform (KS against Uniform(0,1), ). On a sample that "chisq01" rejected at , the bootstrap returns , and on genuinely two-class data the statistic is 17 times the largest of 99 null draws.
Cost. The bootstrap is pairs of refits. Budget accordingly, and use seed for reproducibility.
"LCM_Z" is refused, by both this function and lcsfm(penalty_c = ): it makes the class probabilities depend on covariates, and the penalty is defined on a scalar mixing proportion.
An object of class c("sfa_mlrt", "htest"), so it prints like any other R test. Alongside statistic and p.value it carries null, B, boot (the bootstrap null draws), boot_ok, penalty_c, penalty_at_max, the penalised and unpenalised -class log-likelihoods, logLik_1, class_prob, and reject.
Chen, H., Chen, J. and Kalbfleisch, J.D. (2001). A modified likelihood ratio test for homogeneity in finite mixture models. Journal of the Royal Statistical Society B, 63(1), 19–29.
Stead, A.D., Wheat, P. and Greene, W.H. (2023). On hypothesis testing in latent class and finite mixture stochastic frontier models, with application to a contaminated normal-half normal model. Journal of Productivity Analysis, 60(1), 37–48.
Zhu, H.-T. and Zhang, H. (2004). Hypothesis testing in mixture regression models. Journal of the Royal Statistical Society B, 66(1), 3–16.
set.seed(21) n <- 300 x1 <- rnorm(n) cls <- rbinom(n, 1, 0.5) d <- data.frame(y = ifelse(cls == 1, 4, 1) + x1 + rnorm(n, 0, 0.4) - abs(rnorm(n, 0, 0.6)), x1 = x1) f <- lcsfm(y ~ x1, model_name = "LCM", data = d, n_class = 2) ## Small B for the example only; use the default in real work. lcsfm_homogeneity(f, B = 19, seed = 1, quiet = TRUE)set.seed(21) n <- 300 x1 <- rnorm(n) cls <- rbinom(n, 1, 0.5) d <- data.frame(y = ifelse(cls == 1, 4, 1) + x1 + rnorm(n, 0, 0.4) - abs(rnorm(n, 0, 0.6)), x1 = x1) f <- lcsfm(y ~ x1, model_name = "LCM", data = d, n_class = 2) ## Small B for the example only; use the default in real work. lcsfm_homogeneity(f, B = 19, seed = 1, quiet = TRUE)
For a model whose inefficiency scale depends on covariates (the _Z
models), the marginal effects of those covariates on the mean and variance of
the inefficiency term, and
.
| Fit | Link on |
Covariates reported |
sfm() "NHN_Z", "NE_Z" |
SD (default) or z_link = "var" |
those in z |
sfm() "NTN" with uhet/muhet |
SD (default) or z_link = "var" |
those in either design |
sfm() "NTN" with scaling |
one factor |
those in scaling |
psfm() "TRE_Z", "GTRE_Z" |
variance | those in z (or zp, component = "h")
|
marginal_effects(object, average = FALSE, component = c("u", "h"))marginal_effects(object, average = FALSE, component = c("u", "h"))
object |
an object of class |
component |
which inefficiency component to differentiate. |
average |
if |
A coefficient is not interpretable on its own: it sits in a log
link for a scale parameter, so its units are neither those of nor
those of , and its magnitude is not comparable across models whose link
differs. The marginal effect is on the scale of , and is what applied
work reports.
Writing , every derivative reduces to
times a constant that cancels back into the
moment itself. For the half-normal and the exponential alike:
The truncated normal does not reduce this way. For
model_name = "NTN" the pre-truncation mean enters
alongside the scale, so is not proportional to
and the one-line forms above do not apply. With
and ,
and, since ,
The reported effect is the chain rule through both designs, since
and may depend on different covariates:
(an identity link) and
as above. A covariate appearing in one
design but not the other contributes only through that one, and every
non-constant covariate of either design gets a column, so muhet = ~ z
alone, with a constant , still reports . Note that
is therefore not a semi-elasticity, unlike the
scale coefficients.
The scaling property is exact and needs no family branch. Under
sfm(scaling = ~ z) — implemented for model_name = "NTN" only,
since for a half-normal is
and uhet already fits it — one factor
multiplies both the scale and the
pre-truncation mean. So is constant in ,
, and
whatever the inefficiency family is. Here is the
semi-elasticity , so
dE\_u.dz / E\_u is constant across observations by construction.
The factor of two matters. sfm()'s "NHN_Z" and
"NE_Z" place the linear predictor on the standard deviation,
, while psfm()'s "TRE_Z" and
"GTRE_Z" place it on the variance,
. Reading a from one family
with the other's convention in mind is wrong by exactly this factor. Reporting
the marginal effect instead of the coefficient removes the trap, since the
effect is on the scale of either way.
The link is read from the fit rather than assumed, so a sfm
model fitted with z_link = "var" and a psfm model are
handled the same way and their effects are directly comparable.
psfm(model_name = "GTRE_Z") separates persistent inefficiency
from transient and gives each its own
determinant block, so both are available: component = "u" for the
transient effects and component = "h" for the persistent ones. Each
block is located by NAME rather than by position, because the
coefficients are the trailing ones and a positional rule would report them
under a label.
Effects are per observation, because varies with .
Constant columns of the design are dropped: the derivative with
respect to an intercept is not a marginal effect.
Standard errors are deliberately not reported. The effect is a nonlinear
function of both and , so its sampling distribution
requires the delta method through the full covariance matrix or a bootstrap;
a number printed without one would be read as inference.
With average = FALSE, a data frame with one row per observation:
sigma_u, E_u, Var_u, then one dE_u.d<z> and one
dVar_u.d<z> column per non-constant variance determinant. With
component = "h" the same columns are named sigma_h,
E_h, Var_h, dE_h.d<z> and dVar_h.d<z>.
Attributes "average" (the average marginal effects), "link",
"family" and "component" record the convention used.
With average = TRUE, just the named vector of average marginal effects.
dat <- data_gen_cs(N = 300, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) fit <- sfm(y_pcs_z ~ x1 + x2 | z, data = dat, model_name = "NHN_Z") me <- marginal_effects(fit) head(me) ## The conventional summary. marginal_effects(fit, average = TRUE)dat <- data_gen_cs(N = 300, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) fit <- sfm(y_pcs_z ~ x1 + x2 | z, data = dat, model_name = "NHN_Z") me <- marginal_effects(fit) head(me) ## The conventional summary. marginal_effects(fit, average = TRUE)
The mean efficiency implied by the fitted distribution of inefficiency,
, and the mean efficiency among the most efficient of
that distribution.
meanefficiency(object, p = c(0.01, 0.05, 0.10, 0.25, 0.50, 0.75, 0.90, 0.95, 0.99), use_closed_form = TRUE)meanefficiency(object, p = c(0.01, 0.05, 0.10, 0.25, 0.50, 0.75, 0.90, 0.95, 0.99), use_closed_form = TRUE)
object |
An |
p |
Proportions in |
use_closed_form |
Use the closed-form expression where one exists. Setting
it to |
A different quantity from the reported average. print and
summary report the average of the per-observation conditional
predictions . That is a property of the
sample in hand. is a property of the fitted model, and
it is the number that can be compared across studies.
Closed forms. Available for the half-normal
(), exponential
(), truncated normal, Rayleigh, gamma
(), uniform () and generalized
exponential (). The lognormal, Weibull and
Nakagami have none in elementary terms and are integrated numerically;
method in the returned list says which route was taken.
Every closed form is checked against numerical integration of the same density in the package's tests, and the density is separately checked against its own quantile function by simulation. The first catches an algebraic slip; the second catches a parameter being read off the fit incorrectly, which would otherwise make both routes agree and both be wrong.
Supra-percentile means are always integrated. Efficiency is decreasing
in , so the most efficient are the smallest of :
the reported value is . It necessarily
exceeds the unconditional mean and falls as rises.
A list with model, distribution, the parameters used,
mean_efficiency, the method used to obtain it, and supra,
a data frame of p against mean efficiency.
efficiency for the per-observation predictors.
d <- data_gen_cs(N = 400, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) r <- meanefficiency(fit) r$mean_efficiency r$suprad <- data_gen_cs(N = 400, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) r <- meanefficiency(fit) r$mean_efficiency r$supra
Reports, for each pair of noise and inefficiency distributions, the set of skewness and excess kurtosis values the composed error can attain at all, and says which pairs the residual moments rule out. This is the Type II failure check of Papadopoulos and Parmeter (2021): a pair whose attainable set does not contain the sample moments is refuted without any test statistic. It needs no frontier fit.
moment_range(object = NULL, noise = c("normal", "logistic", "laplace", "uniform", "t"), inefficiency = c("halfnormal", "exponential", "genexponential", "truncatednormal", "gamma"), inefdec = NULL, df = 8)moment_range(object = NULL, noise = c("normal", "logistic", "laplace", "uniform", "t"), inefficiency = c("halfnormal", "exponential", "genexponential", "truncatednormal", "gamma"), inefdec = NULL, df = 8)
object |
An |
noise |
Assumed distributions of |
inefficiency |
Assumed distributions of |
inefdec |
Orientation. Taken from the fit when there is one, otherwise production. A cost frontier puts the skewness on the positive side. |
df |
Degrees of freedom for |
The two equations. With the share of composed-error variance contributed by , Papadopoulos and Parmeter (2021, eqs. 5 and 10) give
Because lies in , the composed skewness can never exceed the skewness of itself, whatever the noise: the Normal-Half-Normal pair is confined to and the Normal-Exponential pair to . The kurtosis equation is quadratic in , so its range is not simply the interval between the two components' values – with leptokurtic noise it dips below both before rising, reaching at .
Which can fail. Half-normal, exponential and generalized exponential have fixed skewness and kurtosis, so their attainable sets are the narrowest. The truncated normal is indexed by and sweeps and – it is the only one here whose composed error can be platykurtic under normal noise. The gamma has and , unbounded as , so a gamma inefficiency term can never suffer a Type II failure; the price, as PP2021 note, is that makes the density log-convex.
This is a diagnostic, not a test. PP2021 section 2.3 shows the range check over-rejects, and the package's own replication of their Tables 1-5 confirms it: with correctly specified Normal-Half-Normal data at and , the sample excess kurtosis falls outside its theoretical range 26% of the time, and the sample skewness 16% of the time. Use spec_test for the formal statistic, which is built on the same two equations but studentised and bootstrapped.
Relation to the wrong-skew problem. A Type I failure – residual skewness of the wrong sign, Waldman (1982) – is a property of the data and is reported for every row at once. Type II failures are pair-specific: they are what tells you that the residuals are too skewed, or too heavy-tailed, for the particular you assumed. Wrong skew is handled by skewness_test and, as an estimator, by esfm.
A data frame of class "sfa_moment_range", one row per pair, with the attainable interval for the composed skewness (g1_lo, g1_hi) and excess kurtosis (g2_lo, g2_hi), logical skew_ok and kurt_ok, and a verdict of "ok", "type I (wrong skew)" or "type II (...)". The sample moments, sample size and orientation are attributes.
Papadopoulos, A. and Parmeter, C. F. (2021). Type II failure and specification testing in the Stochastic Frontier Model. European Journal of Operational Research 293(3), 990-1001.
Waldman, D. M. (1982). A stationary point for the stochastic frontier likelihood. Journal of Econometrics 18(2), 275-279.
spec_test for the formal test built on the same two equations, skewness_test for the Type I failure, esfm for an estimator that tolerates it.
## The reference table on its own: no data needed. moment_range(noise = "normal") set.seed(4) d <- data_gen_cs(N = 400, rand = 4, sig_u = 1, sig_v = 0.5, cons = 1, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) ## Exponential inefficiency, read through a normal-half-normal lens: the ## residuals are more skewed than a half-normal can make them. fit <- sfm(y_pcs_e ~ x1 + x2, data = d, model_name = "NHN") moment_range(fit, noise = "normal")## The reference table on its own: no data needed. moment_range(noise = "normal") set.seed(4) d <- data_gen_cs(N = 400, rand = 4, sig_u = 1, sig_v = 0.5, cons = 1, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) ## Exponential inefficiency, read through a normal-half-normal lens: the ## residuals are more skewed than a half-normal can make them. fit <- sfm(y_pcs_e ~ x1 + x2, data = d, model_name = "NHN") moment_range(fit, noise = "normal")
Fits a stochastic frontier whose frontier is estimated by kernel methods rather than assumed linear. Five methods are available:
"FLW" (the default): kernel regression, then constant scale parameters from the residuals. Fast and steady.
"SVKZ": local moments, so and vary with the covariates. No optimizer.
"PSZ": local maximum likelihood, with varying and . Slow.
"MY": iterative local likelihood, with constant scale parameters. Slow, and the most accurate in the comparison under Details.
"SZ": a DEA step that makes an existing smooth frontier monotone, and under rts = "vrs" or "crs" also concave.
Only "FLW" offers inefficiency distributions other than the half-normal. The result is an "npsfareg" object, with no coefficient vector or standard errors.
npsfm(formula, data, method = c("FLW", "SVKZ", "PSZ", "KPST", "MY", "SZ"), dist = c("hn", "exp", "gamma", "unif"), regtype = c("lc", "ll"), bw.sel = c("cv.ls", "cv.aic"), bw = NULL, cost = FALSE, eff = TRUE, maxit = 5000, tol = 1e-3, iter = 25, rts = c("vrs", "crs", "drs", "irs"), prior.fit = NULL, log.form = TRUE, verbose = FALSE)npsfm(formula, data, method = c("FLW", "SVKZ", "PSZ", "KPST", "MY", "SZ"), dist = c("hn", "exp", "gamma", "unif"), regtype = c("lc", "ll"), bw.sel = c("cv.ls", "cv.aic"), bw = NULL, cost = FALSE, eff = TRUE, maxit = 5000, tol = 1e-3, iter = 25, rts = c("vrs", "crs", "drs", "irs"), prior.fit = NULL, log.form = TRUE, verbose = FALSE)
formula |
A single-part symbolic description of the frontier, |
data |
A data frame containing the variables named in |
method |
Which estimator to use.
Matching ignores case, so |
dist |
Distribution of the one-sided inefficiency term, for |
regtype |
Kernel regression type passed to |
bw.sel |
Bandwidth selection method: |
bw |
Optional numeric vector of bandwidths, one per covariate. When supplied, cross-validation is skipped and these are used directly – useful for undersmoothing, for sensitivity analysis, or simply to avoid paying for bandwidth selection repeatedly in a simulation. |
cost |
Logical. |
eff |
Logical. Compute observation-level inefficiency predictions ( |
maxit |
Maximum function evaluations for each local optimization under |
tol |
Convergence tolerance for |
iter |
Maximum number of outer iterations for |
rts |
Returns-to-scale assumption for the DEA step under |
prior.fit |
For |
log.form |
For |
verbose |
Logical. Report progress through the per-observation loops of |
All five methods keep the composed-error structure of sfm and relax only the linear frontier.
A least-squares kernel regression of on estimates , not . "FLW" and "SVKZ" start from that regression and shift it up by an estimate of . For "FLW" the shift is one number, because is constant, and the frontier gradients are those of the conditional mean. For "SVKZ", comes from the local third moment, so the shift varies across observations and the gradient picks up a chain-rule term.
"PSZ" and "MY" instead maximize the composed-error likelihood locally. The local intercept is then itself, and no shift is applied. The cost is one numerical optimization per observation (for "MY", per observation per iteration), so expect them to be one to two orders of magnitude slower than "FLW". Both start from an "FLW" fit.
On a simulated nonlinear frontier with , (3 replications, ), mean absolute frontier error was 0.070 for "MY", 0.079 for "FLW", and 0.116 for both "SVKZ" and "PSZ". "FLW" is the steadiest for its cost, and "MY" the most accurate if the run time is acceptable. "SVKZ" and "PSZ" are for the case where really varies with , which this design does not test.
is identified by the residuals being negatively skewed. "FLW" inverts a single sample moment and either succeeds or warns. "SVKZ" inverts a local third moment, which is much noisier. Wherever its sign is wrong the implied is negative, and, following the paper, is floored at 0 there and that point's contribution to the frontier gradient is set to zero. wrong.skew flags these observations. If many are flagged, the local third moment is too noisy to inform the fit, and "FLW" is the safer choice at moderate sample sizes.
Kernel regression and bandwidth selection come from the np package, and "SZ"'s DEA step from the DEA package. Both are in Suggests, and npsfm() stops with an install instruction if the one it needs is missing. Cross-validated bandwidth selection dominates the run time and scales quadratically in the sample size; supply bw to skip it.
An object of class "npsfareg". This is deliberately not an "sfareg" object: there is no parameter vector with standard errors, so coef(), vcov() and logLik() would have nothing meaningful to return. fitted(), residuals(), nobs(), print() and summary() are provided. A component that a method does not produce is absent from the list, not NULL. Components:
frontier |
The estimated frontier |
frontier.grad |
Matrix of estimated frontier gradients, one row per observation and one column per covariate. Under |
conditional.mean |
The implied |
residuals |
Composed residuals measured against the corrected frontier, |
mean.correction |
The estimated |
sigma.u, sigma.v
|
Estimated scale parameters. Scalars under |
lambda, sigma
|
Returned by |
theta |
Returned by |
b |
Returned by |
sigma.u.grad, wrong.skew
|
Returned by |
convergence |
Returned by |
iterations, converged, tol.reached
|
Returned by |
prior.fit, dea.efficiency, rts
|
Returned by |
u_hat, exp_u_hat
|
Jondrow et al. (1982) inefficiency predictions |
bws |
The bandwidth object(s) used: one for |
method, dist, formula, call, cost, regtype, bw.sel, nobs, total_time, data
|
Settings and bookkeeping. |
Requires the np package (and DEA for "SZ"), both in Suggests.
Christopher F. Parmeter and David H. Bernstein
Fan, Y., Li, Q. and Weersink, A. (1996) 'Semiparametric estimation of stochastic production frontier models', Journal of Business & Economic Statistics, 14(4), pp. 460-468.
Simar, L., Van Keilegom, I. and Zelenyuk, V. (2017) 'Nonparametric least squares methods for stochastic frontier models', Journal of Productivity Analysis, 47(3), pp. 189-204.
Kumbhakar, S.C., Park, B.U., Simar, L. and Tsionas, E.G. (2007) 'Nonparametric stochastic frontiers: a local maximum likelihood approach', Journal of Econometrics, 137(1), pp. 1-27.
Park, B.U., Simar, L. and Zelenyuk, V. (2015) 'Categorical data in local maximum likelihood: theory and applications to productivity analysis', Journal of Productivity Analysis, 43(2), pp. 199-214.
Martins-Filho, C. and Yao, F. (2015) 'Semiparametric stochastic frontier estimation via profile likelihood', Econometric Reviews, 34(4), pp. 413-451.
Simar, L. and Zelenyuk, V. (2011) 'Stochastic FDH/DEA estimators for frontier analysis', Journal of Productivity Analysis, 36(1), pp. 1-20.
Jondrow, J., Lovell, C.A.K., Materov, I.S. and Schmidt, P. (1982) 'On the estimation of technical inefficiency in the stochastic frontier production function model', Journal of Econometrics, 19(2-3), pp. 233-238.
Battese, G.E. and Coelli, T.J. (1988) 'Prediction of firm-level technical efficiencies with a generalized frontier production function and panel data', Journal of Econometrics, 38(3), pp. 387-399.
sfm for parametric cross-sectional frontiers, psfm for panel models, and data_gen_cs for simulating data with known true parameters.
if (requireNamespace("np", quietly = TRUE)) { set.seed(42) n <- 150 x1 <- runif(n, 1, 4) x2 <- runif(n, 1, 4) m <- 1 + 0.6 * log(x1) + 0.4 * sqrt(x2) # nonlinear frontier d <- data.frame(y = m + rnorm(n, 0, 0.25) - abs(rnorm(n, 0, 0.6)), x1 = x1, x2 = x2) ## Fan, Li and Weersink, normal-half-normal f <- npsfm(y ~ x1 + x2, data = d, method = "FLW", dist = "hn") f head(fitted(f)) head(f$exp_u_hat) ## Simar, Van Keilegom and Zelenyuk: sigma_u and sigma_v vary with x g <- npsfm(y ~ x1 + x2, data = d, method = "SVKZ") summary(g$sigma.u) }if (requireNamespace("np", quietly = TRUE)) { set.seed(42) n <- 150 x1 <- runif(n, 1, 4) x2 <- runif(n, 1, 4) m <- 1 + 0.6 * log(x1) + 0.4 * sqrt(x2) # nonlinear frontier d <- data.frame(y = m + rnorm(n, 0, 0.25) - abs(rnorm(n, 0, 0.6)), x1 = x1, x2 = x2) ## Fan, Li and Weersink, normal-half-normal f <- npsfm(y ~ x1 + x2, data = d, method = "FLW", dist = "hn") f head(fitted(f)) head(f$exp_u_hat) ## Simar, Van Keilegom and Zelenyuk: sigma_u and sigma_v vary with x g <- npsfm(y ~ x1 + x2, data = d, method = "SVKZ") summary(g$sigma.u) }
The dataset is a cross-section of U.S. commercial banks for 1989, extracted from the panel dataset used by Kumbhakar, Parmeter and Tsionas (2013) and based on the Federal Reserve Bank of Chicago's Reports of Condition and Income. It contains detailed cost data with inputs and outputs defined under the intermediation approach, and input prices constructed as expense-quantity ratios.
data("panel89")data("panel89")
A data frame with 4,985 observations on the following 11 variables.
ya numeric vector containing the natural logarithm of total cost*
q1a numeric vector containing the natural logarithm of installment loans
q2a numeric vector containing the natural logarithm of real estate loans
q3a numeric vector containing the natural logarithm of business loans
q4a numeric vector containing the natural logarithm of federal funds sold and securities purchased
q5a numeric vector containing the natural logarithm of other assets
w1a numeric vector containing the natural logarithm of the price of labour*
w2a numeric vector containing the natural logarithm of the price of capital*
w3a numeric vector containing the natural logarithm of the price of purchased funds*
w4a numeric vector containing the natural logarithm of the price of interest-bearing deposits in total transaction accounts*
za numeric vector containing the natural logarithm of total assets
*The cost and input price variables are normalised by that of a fifth input: the price of interest-bearing deposits in total non-transaction accounts. Total cost is defined as the sum of total expenses for each input. Input prices are derived by dividing the total expense for each input by the corresponding input quantity.
Kumbhakar, S.C., Parmeter, C.F. and Tsionas, E.G. (2013) 'A zero inefficiency stochastic frontier model', Journal of Econometrics, 172(1), pp. 66-76. doi:10.1016/j.jeconom.2012.08.021.
Kumbhakar, S.C. and Tsionas, E.G. (2005) 'Measuring technical and allocative inefficiency in the translog cost system: a Bayesian approach', Journal of Econometrics, 126(2), pp. 355-384. doi:10.1016/j.jeconom.2004.05.006.
data(panel89) plot(panel89)data(panel89) plot(panel89)
Density and distribution function of the stochastic frontier composed error (production) or (cost), with and independent. The density is a skew-normal and closed form; the distribution function is not, and is obtained here by quadrature that stays accurate far into the tails.
dcomposed(x, sigma_u, sigma_v, inefdec = TRUE, log = FALSE) pcomposed(q, sigma_u, sigma_v, inefdec = TRUE, lower.tail = TRUE, log.p = FALSE, method = c("quadrature", "simulate"), n_nodes = 128, R = 1e6, seed = NULL)dcomposed(x, sigma_u, sigma_v, inefdec = TRUE, log = FALSE) pcomposed(q, sigma_u, sigma_v, inefdec = TRUE, lower.tail = TRUE, log.p = FALSE, method = c("quadrature", "simulate"), n_nodes = 128, R = 1e6, seed = NULL)
x, q
|
Numeric vector of quantiles. |
sigma_u |
Standard deviation of the pre-truncation normal behind |
sigma_v |
Standard deviation of the noise, strictly positive. |
inefdec |
|
log, log.p
|
Return the log of the density or probability. Use |
lower.tail |
If |
method |
|
n_nodes |
Quadrature nodes, at least 16. |
R, seed
|
Draws and optional seed for |
Why this exists. Every likelihood in sfa evaluates the composed error's density and none needs its distribution function, which is why there was not one. Copula models need it: a Gaussian copula on the composed error contains , so an that saturates at 0 or 1 does not merely lose precision, it returns an infinity and takes the copula density with it.
How it is computed. Conditioning on leaves a normal distribution function in closed form, so
with for a production frontier and for a cost frontier. The noise is therefore never drawn or integrated, and the tails inherit the accuracy of pnorm, which is reliable twenty standard deviations out. The expectation over is taken by Gauss-Legendre quadrature on , and accumulated by log-sum-exp so that terms which would underflow individually still contribute.
The upper tail. Requesting lower.tail = FALSE negates the argument of rather than forming . At with the complement is exactly 1 in double precision while the direct computation returns a finite log-probability below .
Accuracy. There is no closed form except at equal to 0 (the normal) or 1, where Amsler, Schmidt and Tsay show for the cost case. Against that standard the default quadrature is accurate to about relative at probabilities as small as . It is deliberately used everywhere rather than special-cased at : the production form of that identity is a complement, which would reintroduce the saturation the function exists to avoid.
A numeric vector the length of x or q.
Amsler, C., Schmidt, P. and Tsay, W.-J. (2019). Evaluating the cdf of the distribution of the stochastic frontier composed error. Journal of Productivity Analysis, 52(1), 29–35.
## Density integrates to one. integrate(function(x) dcomposed(x, 1, 0.5), -Inf, Inf)$value ## Distribution function, and a tail that does not saturate. pcomposed(c(-2, 0, 2), sigma_u = 1, sigma_v = 0.5) pcomposed(30, 1, 1, lower.tail = FALSE, log.p = TRUE) ## Checked against the one exact standard there is (cost frontier). su <- sv <- sqrt(0.5) c(exact = pnorm(-6 / (sqrt(2) * su))^2, computed = exp(pcomposed(-6, su, sv, inefdec = FALSE, log.p = TRUE)))## Density integrates to one. integrate(function(x) dcomposed(x, 1, 0.5), -Inf, Inf)$value ## Distribution function, and a tail that does not saturate. pcomposed(c(-2, 0, 2), sigma_u = 1, sigma_v = 0.5) pcomposed(30, 1, 1, lower.tail = FALSE, log.p = TRUE) ## Checked against the one exact standard there is (cost frontier). su <- sv <- sqrt(0.5) c(exact = pnorm(-6 / (sqrt(2) * su))^2, computed = exp(pcomposed(-6, su, sv, inefdec = FALSE, log.p = TRUE)))
for the composed error of any cross-sectional
model, where pcomposed covers only the half-normal case.
pcomposed_model(q, model_name, par, inefdec = TRUE, lower.tail = TRUE, log.p = FALSE, n_nodes = 128) composed_cdf(object, q = NULL, data = NULL, ...)pcomposed_model(q, model_name, par, inefdec = TRUE, lower.tail = TRUE, log.p = FALSE, n_nodes = 128) composed_cdf(object, q = NULL, data = NULL, ...)
q |
Numeric vector of quantiles. For |
model_name |
One of |
par |
A named parameter vector, as in |
inefdec |
|
lower.tail, log.p
|
As elsewhere in R. |
n_nodes |
Quadrature nodes over the inefficiency term. |
object |
An |
data |
The data the model was fitted to. |
... |
Passed to |
The probability is computed as an expectation over the inefficiency term,
with for production and for cost. Conditioning on
leaves the noise CDF in closed form, so the noise is never integrated
and the tails inherit pnorm()'s (or pt()'s) own accuracy.
Three parameterizations do not read the way their names suggest, and are
taken from each likelihood in sfm() rather than from the label:
"THT" lists sigu before sigv, the only model that
does; "NG"'s sigu is the gamma scale and its mu
the shape; "NNAK"'s sigu is the Nakagami spread, so
; and "NLN"'s mu is a
meanlog.
"THT" is not an independent convolution. Tancredi's (2002)
composed error is skew-, a scale mixture in which and
are divided by the same . Treating it as
noise plus an independent half-normal – which is what "tHN"
actually is – gets the log density wrong by up to 1.74, so the two models
take different paths here despite looking alike.
Verified two ways: against pcomposed for the half-normal case,
agreeing to relative including a lower tail at
; and for all thirteen models against a four-million-draw
simulation, with a maximum absolute error of against a
Monte Carlo standard error of . Numerically
differentiating this CDF reproduces each model's own stored log-density to
about , except for the simulated-ML models "NLN" and
"NW", where the difference is the simulation error of their
likelihood rather than a disagreement about the model.
A numeric vector of probabilities (or their logs).
p <- c(sigv = 0.5, sigu = 1.1) pcomposed_model(c(-2, -1, 0, 1), "NE", p) d <- as.data.frame(data_gen_cs(N = 200, rand = 5, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1)) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) head(composed_cdf(f, data = d))p <- c(sigv = 0.5, sigu = 1.1) pcomposed_model(c(-2, -1, 0, 1), "NE", p) d <- as.data.frame(data_gen_cs(N = 200, rand = 5, sig_u = 1, sig_v = 0.5, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1)) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) head(composed_cdf(f, data = d))
Notes on psfm(model_name = "PL80_MVTN"), which fits the likelihood derived in Appendix 2 of Pitt and Lee (1981) but never used there.
The model is
with the firm's inefficiency vector drawn from a -variate normal truncated to the negative orthant, and iid independent of . Unlike model_name = "PL80", which holds inefficiency fixed over time, here it varies across periods and is correlated within a firm – carries that dependence.
Why it was never used. Pitt and Lee derived this likelihood and then set it aside, writing that it “is difficult to evaluate since the quantities and involve -dimensional numerical integrals”, and estimating Model III by Zellner seemingly-unrelated regression instead. Those quantities are orthant probabilities of a multivariate normal. mnormt::sadmvn() evaluates one in about 3 milliseconds at , so a likelihood evaluation costs roughly of them – about 0.3 s at , and about 12 s for a whole fit at , . What was intractable in 1981 is merely slow now.
Writing and , the per-firm log density is
where for is the truncation constant and for .
is equicorrelated, not unrestricted. It is parameterized as , costing two parameters. An unrestricted costs – 21 at , 55 at – on top of and , every one identified only through orthant probabilities. The equicorrelated form captures what the general was introduced for: dependence of a firm's inefficiency across periods. gives inefficiency independent over time; approaches the time-invariant "PL80" case. Because the form is equicorrelated every matrix quantity above is closed form (Sherman-Morrison), so only the orthant probabilities are numerical.
Requirements and limits. A balanced panel with : is a single matrix shared by every firm, and with there is no cross-period dependence for it to describe. An unbalanced panel is an error pointing at "PL80". Reported parameters are sigv, sigu, rho and the frontier coefficients.
u_hat is the posterior mean of from the Gaussian part, floored at zero, not : the exact conditional mean of a truncated multivariate normal is another -dimensional integral. It is exact where the truncation does not bind and an approximation where it does.
David H. Bernstein
Pitt, M.M. and Lee, L.-F. (1981) 'The measurement and sources of technical inefficiency in the Indonesian weaving industry', Journal of Development Economics, 9(1), pp. 43-64. See Appendix 2 for the likelihood.
psfm, data_gen_p for the y_pl_mvtn column that tests it.
library(sfa) d <- data_gen_p(t = 4, N = 60, rand = 5, sig_u = 1, sig_v = 0.3, sig_r = 0.2, sig_h = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) f <- psfm(y_pl_mvtn ~ x1 + x2, model_name = "PL80_MVTN", data = as.data.frame(d), individual = "name") f$outlibrary(sfa) d <- data_gen_p(t = 4, N = 60, rand = 5, sig_u = 1, sig_v = 0.3, sig_r = 0.2, sig_h = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) f <- psfm(y_pl_mvtn ~ x1 + x2, model_name = "PL80_MVTN", data = as.data.frame(d), individual = "name") f$out
Standard modelling generics for objects of class "sfareg", as returned by sfm, psfm, zsfm and ttsfm.
## S3 method for class 'sfareg' predict(object, newdata = NULL, type = c("frontier", "response", "efficiency"), ...) ## S3 method for class 'sfareg' fitted(object, ...) ## S3 method for class 'sfareg' residuals(object, ...)## S3 method for class 'sfareg' predict(object, newdata = NULL, type = c("frontier", "response", "efficiency"), ...) ## S3 method for class 'sfareg' fitted(object, ...) ## S3 method for class 'sfareg' residuals(object, ...)
object |
An object of class |
newdata |
Optional data frame containing the frontier regressors at which to predict. If omitted, the data the model was fitted to is used. |
type |
|
... |
Ignored. |
Only the first part of a pipe formula (y ~ x | z | zp) describes the frontier; later parts parameterize variances and take no part in prediction.
Frontier coefficients are identified by matching the fitted coefficient names against the frontier design matrix, so these methods work for every model in the package without depending on where a given model happens to place its auxiliary parameters (lambda/sigma for NHN, sigv/sigu for NE, sigmaSq/gamma for PL80, and so on).
type = "efficiency" and type = "response" are available only for the estimation sample, since both require the composed residual and therefore the response; requesting them with newdata raises an error rather than returning something misleading. "efficiency" additionally requires a model that returns an efficiency prediction – see sfm for which ones do.
A numeric vector, one element per observation.
David H. Bernstein
library(sfa) d <- data_gen_cs(N = 200, rand = 1, sig_u = 0.8, sig_v = 0.3, cons = 1, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) head(fitted(f)) head(residuals(f)) head(predict(f, type = "efficiency")) head(predict(f, newdata = d[1:5, ]))library(sfa) d <- data_gen_cs(N = 200, rand = 1, sig_u = 0.8, sig_v = 0.3, cons = 1, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) head(fitted(f)) head(residuals(f)) head(predict(f, type = "efficiency")) head(predict(f, newdata = d[1:5, ]))
Compact display of a stochastic frontier fit, as returned by sfm, psfm, zsfm or ttsfm.
## S3 method for class 'sfareg' print(x, ...)## S3 method for class 'sfareg' print(x, ...)
x |
An object of class |
... |
Additional arguments passed to other methods |
Prints the model name, the estimated coefficients with their standard errors and t-values (the out matrix), and the log-likelihood where the estimator has one. For a fuller display, including the frontier and variance-parameter blocks, use summary.sfareg.
No return value, called for side effects
David H. Bernstein
summary.sfareg, sfareg-methods
library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) print(cs.nhnz)library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) print(cs.nhnz)
Function to implement various panel data stochastic frontier estimators
psfm(formula, model_name = c("TRE_Z", "GTRE_Z", "TRE", "GTRE", "GTRE_FML", "TFE", "TFE_WMLE", "FD", "GTRE_SEQ1", "GTRE_SEQ2", "SSFE", "PL80", "PL80_MVTN", "BC92", "K1990", "K1990modified", "CSS", "LS", "SSRE", "SSCRE", "KSS"), data, maxit.bobyqa = 5000, maxit.nlminb = 500, maxit.psoptim = 100, maxit.optim = 1000, REPORT = 1, trace = 3, pgtol = 0, individual, halton_num = NULL, start_val = FALSE, gamma = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, Method = "L-BFGS-B", verbose = FALSE, rand.gtre = NULL, rand.psoptim = NULL, OPG_calc = FALSE, estimator = c("fiml", "sml", "seq1", "seq2"), collinear_action = c("start_only", "error", "warn_drop"), time = NULL, tfe_lambda_max = 100, keep_objective = FALSE, kss_L = "auto", kss_smooth = "auto", kss_L_max = 7L, mundlak = NULL, sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, sim_burn = NULL, sim_scrambling = 0L, sim_prime = NULL)psfm(formula, model_name = c("TRE_Z", "GTRE_Z", "TRE", "GTRE", "GTRE_FML", "TFE", "TFE_WMLE", "FD", "GTRE_SEQ1", "GTRE_SEQ2", "SSFE", "PL80", "PL80_MVTN", "BC92", "K1990", "K1990modified", "CSS", "LS", "SSRE", "SSCRE", "KSS"), data, maxit.bobyqa = 5000, maxit.nlminb = 500, maxit.psoptim = 100, maxit.optim = 1000, REPORT = 1, trace = 3, pgtol = 0, individual, halton_num = NULL, start_val = FALSE, gamma = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, Method = "L-BFGS-B", verbose = FALSE, rand.gtre = NULL, rand.psoptim = NULL, OPG_calc = FALSE, estimator = c("fiml", "sml", "seq1", "seq2"), collinear_action = c("start_only", "error", "warn_drop"), time = NULL, tfe_lambda_max = 100, keep_objective = FALSE, kss_L = "auto", kss_smooth = "auto", kss_L_max = 7L, mundlak = NULL, sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, sim_burn = NULL, sim_scrambling = 0L, sim_prime = NULL)
formula |
a symbolic description for the model to be estimated |
model_name |
Which panel estimator to fit. All are described in ‘Details’; in brief:
Matching ignores case, so |
data |
A panel data set. This may be an ordinary |
maxit.bobyqa |
Maximum number of iterations for the bobyqa optimization routine |
maxit.nlminb |
Maximum number of iterations for the nlminb optimization routine. |
maxit.psoptim |
Maximum number of iterations for the psoptim optimization routine |
maxit.optim |
Maximum number of iterations for the optim optimization routine |
REPORT, trace, pgtol
|
Accepted but not used by |
individual |
individual unit in the regression model |
halton_num |
Number of simulation draws per firm for the simulated-ML models ( |
start_val |
starting value (optional) |
gamma |
Logical. Report the variance parameters in the |
PSopt |
use psoptim optimization routine (T or F) |
optHessian |
Logical. Should a numerically differentiated Hessian matrix be returned while using the optim routine? (for optim routine) |
inefdec |
Production or cost function |
Method |
Method passed to |
verbose |
Logical. Print optimization progress messages? Default is |
rand.psoptim |
Integer. Seed for replication of psoptim. Default to |
rand.gtre |
Integer. Seed for replication of the gtre model. Default to |
OPG_calc |
Logical. Should the outer product of gradients be calculated for inference. Default to |
estimator |
How to estimate the four-component GTRE model, in the same spirit as
The older names remain and are unchanged: On an unbalanced panel |
collinear_action |
One of |
time |
Character. Name of a time-period column in |
tfe_lambda_max |
Numeric. Upper bound placed on |
keep_objective |
Retain the likelihood closure on the fitted object as |
kss_L |
|
kss_smooth |
|
kss_L_max |
|
sim_type |
Sequence used to build the draws for the simulated-ML models listed under |
antithetics |
Logical. If |
sim_burn |
Number of leading elements of the sequence discarded. Default |
sim_scrambling |
Integer 0–3, |
sim_prime |
Prime for |
mundlak |
Mundlak adjustment terms: a formula naming the regressors whose within-firm means should be added to the frontier, or |
The generalized true random effects model (GTRE, 4-component model) and true random effects models (TRE) are both estimated by simulated maximum likelihood based on the paper by the Fillipini and Greene (2016, JPA). The TRE_Z and GTRE_Z allow for modeling the u-component of the GTRE and TRE with determinants of inefficiency. The first-difference estimator (FD) of Wang and Ho (2010, JoE) is also available.
model_name |
what it models | use it when |
"TRE", "TRE_Z" |
true random effects: firm heterogeneity plus transient inefficiency | firm effects are not inefficiency |
"GTRE", "GTRE_Z" |
four components: heterogeneity, persistent and transient inefficiency | both kinds of inefficiency matter |
"GTRE_SEQ1", "GTRE_SEQ2" |
the four-component model in two steps | is large; inconsistent for fixed |
"TFE", "TFE_WMLE" |
true fixed effects | firm effects may be correlated with the regressors |
"FD" |
first differences (Wang and Ho 2010) | fixed effects with inefficiency determinants |
"PL80", "BC92", "PL80_MVTN" |
time-invariant, decaying or correlated inefficiency | the classical panel frontiers |
"K1990", "K1990modified" |
Kumbhakar (1990) time paths for inefficiency | inefficiency follows a smooth time pattern |
"SSFE", "SSRE", "SSCRE" |
Schmidt-Sickles: the firm effect is the inefficiency | no distributional assumption on |
"CSS", "LS", "KSS" |
distribution-free, time-varying firm effects | inefficiency changes over time, no distribution |
Each family has its own subsection below.
Choosing how GTRE is estimated. Since version 1.1.4 the four routes to the four-component model are selected with estimator rather than by picking among four model_name values that looked like four different models. model_name = "GTRE" now defaults to estimator = "fiml"; through 1.1.3 it fitted the simulated-ML estimator, which is estimator = "sml". Scripts that pass "GTRE" therefore get a different estimator than they did and are warned once per call. The explicit names "GTRE_FML", "GTRE_SEQ1" and "GTRE_SEQ2" are unaffected.
Two ways of estimating the same four-component model. "GTRE" and "GTRE_FML" both fit
with two-sided and one-sided (half-normal), separating persistent inefficiency from transient inefficiency . They differ in how the likelihood is obtained.
"GTRE" uses simulated maximum likelihood over Halton draws. "GTRE_FML" uses full information maximum likelihood through the model's closed-skew-normal representation: with , and , the firm-level density is available in closed form and nothing is integrated by Monte Carlo. The likelihood is therefore deterministic, and repeated fits to the same data give identical answers.
The -dimensional normal CDF this requires would normally be the bottleneck, but its covariance is diagonal-plus-rank-one for this model, so it collapses to a one-dimensional Gauss-Hermite integral – the same reduction used for "TFE".
"GTRE_FML" reports raw standard deviations sigr, sigv, sigh, sigu (individual effect, noise, persistent inefficiency, transient inefficiency), not the / reparameterization "GTRE" uses, and it requires a balanced panel – the closed-skew-normal pieces are built at a single . It errors rather than silently mis-estimating if the panel is unbalanced; use "GTRE" in that case.
Starting values for "GTRE_FML". The closed-skew-normal likelihood carries a boundary optimum at , where the model collapses to "TRE" and the intercept absorbs the missing . Two different things can put a fit there, and the likelihood tells them apart. A fit whose boundary solution has a lower likelihood than the true parameter vector reached it by getting stuck, and is an optimizer failure; one whose boundary solution has a higher likelihood reached it because that is where the maximum is, and is the correct answer for that sample (see ‘When a persistent scale collapses to zero’ below). The start search addresses only the first kind. Following Colombi (2010) and Colombi, Martini and Vittadini (2011), who recommend seeding this likelihood from the two-step moment estimates, psfm() builds two candidate starts – the random-effects decomposition shared with "GTRE", and the two-step estimator reported by "GTRE_SEQ2" – evaluates the likelihood at each, and begins from the better one. Which was chosen, and the log-likelihood at both, is returned in start_search. Supplying start_val overrides the search.
Note that and enter the density only through ; their separation rests entirely on the skewness term. That pair is therefore the least precisely estimated part of the model even when the fit is at the global optimum, which is consistent with the confounding between latent heterogeneity and persistent inefficiency reported in the source papers.
"GTRE_SEQ1" and "GTRE_SEQ2" are large-T estimators. Both take
the random-effects decomposition from plm and hand its two output series to a
second stage – "GTRE_SEQ1" fits an intercept-only normal–half-normal likelihood
to each, "GTRE_SEQ2" inverts their second and third moments. The second stage
treats those series as if they were draws from and ,
and at finite they are not: alpha_hat is the shrunken predictor
with ,
and eps_hat is the quasi-demeaned residual
with
.
Relative to the latent draws this attenuates the third central moment on the
side by (about 0.81 at
) and scales the side by and . None of these
factors tends to 1 as grows with fixed, so both estimators
converge to the wrong constants and are inconsistent for fixed ; each call
warns accordingly. On the package's own convergence design (,
, ) the estimates settle at
against 0.917 and against
0.80, and the log MSE-on-log slopes are near zero rather than . The
same routines applied to the latent draws themselves recover the truth at the
rate, which locates the problem in the first-stage decomposition rather
than in either second stage. Raising removes the bias; raising only
sharpens it. For a consistent alternative use model_name = "GTRE", which
estimates all four variance components jointly.
The standard errors these two report are delta-method errors built from the
iid moment variances, so they describe sampling variability around each
estimator's probability limit, not around the truth. They are also optimistic on the
side, whose residuals are correlated within firm.
When a persistent scale collapses to zero. GTRE carries two persistent components – , persistent inefficiency, and , the firm random effect – and the likelihood cannot always separate them in a given sample. When it cannot, it merges them: one scale goes to zero and the other absorbs its variation, coming back inflated.
This is common rather than exceptional. On simulated data at with both components genuinely present (, ), one of the two collapsed in 37% of 87 replications, and the commoner direction was rather than . The share with both interior does improve with (15, 19 and 21 of about 29 at ) but slowly.
A collapse is usually the right answer, not a numerical failure. On one such replication – whose realized scale was 0.3695, so persistent inefficiency was certainly present – the boundary solution had a log-likelihood higher than the true parameter vector by 3.66, a likelihood ratio of about 39. This is the panel counterpart of the cross-sectional wrong-skewness result (Waldman 1982), where is the correct maximum likelihood estimate rather than an optimizer failure; see skewness_test. This is not in tension with the start search described above under "GTRE_FML": a boundary fit that beats the truth on likelihood is the estimate, while one that loses to it is a trapped optimizer, and the search exists to remove the second so that what remains is the first.
What a collapse does to the rest of the fit, exactly. One rule covers it: a collapse conserves the persistent variance and relocates the persistent mean into the intercept. Those are two different destinations. is half-normal, so it carries both a mean and a variance , and the model has a separate place for each: the frontier intercept takes the mean, since , and the surviving persistent scale takes the variance. The intuitive shortcut – that the survivor comes back at – ignores that split and is wrong.
| surviving scale | intercept | |
|
|
shifts by |
|
|
shifts by
|
At , that predicts 0.3133 and in the first row, 0.5197 and in the second, with nothing fitted. Over 1000 replications at each of five sample sizes the measurements are 0.2973, , 0.5058 and – within 2.5% to 5.1%, and all four slightly below prediction, which is what selection produces rather than a wrong constant.
The practical consequence is that a collapsed fit does not merely mislabel the persistent split. It displaces the intercept by a known and substantial amount, so any level read off coef() is displaced with it.
What a collapse is telling you about your data. It is wrong skewness, measured between firms rather than within them. Take the OLS residuals, average them within each firm – that average is the persistent part of the error – and look at their skew. Over 1000 replications, the samples whose between-firm skew came out positive (the wrong way for a production frontier) lost to the boundary 69% of the time, against 3.6% for the rest; the area under the ROC curve for that single statistic is 0.955. Not one wrong-skew sample collapsed instead. So a collapsed is not a numerical accident to be retried with different starting values: this sample carries no between-firm evidence of persistent inefficiency, and the estimate is reporting that.
Note that this is a reading of the sign, not a test. The firm mean of the residuals contains as well as , and the mean of half-normals is itself skewed the same way, so on short panels the between-firm skew leans negative even when – 82% of the time at , 63% at , 53% at . skewness_test is therefore not applicable to the between-firm residuals as it stands.
Whether firms or periods help depends on the estimator, and the two answers are opposite. Crossed designs on the same data-generating process, fitting the collapse rate on and :
| estimator | fits | log N | log T |
"fiml" (the default) |
600 | -1.66 (0.18) | -0.93 (0.20) |
"sml" |
251 | -0.68 (0.26) | -1.07 (0.27) |
Under the default, firms buy about 1.8 times what periods do; under simulated ML it is the other way round, periods buying about 1.6 times what firms do. Restricting the "fiml" design to , cell for cell the same grid the "sml" one used, leaves it at , so this is the estimator and not the design; the difference in the coefficient is significant and the difference in the coefficient is not.
The practical reading, for the estimator you get by default: add firms. At the collapse rate falls 68%, 36%, 18%, 6% across , and reaches 0% at . Under "sml" the same row is flat – 60%, 39%, 41% – which is why a short panel really is a dead end there however many firms it has. If you must use estimator = "sml" (an unbalanced panel forces it), then length is what buys the split, and a four-period panel will not give it to you.
How to tell in advance whether your panel can support the split. One
quantity governs it, and it is not . What the likelihood has to do is read the persistent part out of firm means that the transient block contaminates, so write the firm-mean composite as and compare its persistent variance with the transient variance left in it:
Across six designs varying all four scales in different combinations (50 replications each, , ), this orders the collapse rate perfectly – Spearman under both estimators, and a logistic coefficient on of :
| S/N | collapse | |
|
|
| 0.67 | 54% | 0.14 | 0.40 | |
| 0.90 | 50% | 0.10 | 0.40 | |
| 0.96 | 40% | 0.21 | 0.19 | |
| 1.30 | 31% | 0.15 | 0.17 | |
| 5.20 | 8% | 0.06 | 0.02 | |
| 11.69 | 6% | 0.04 | 0.02 |
As a rule of thumb: below about half your fits will merge the two components, and it takes roughly to get under 10%.
Two things follow that are easy to get backwards. First, raising does not help – it only decides which scale dies. Going from a ratio of 2 to 4 by shrinking takes down (0.15 to 0.10) and up (0.17 to 0.40), and the net rate rises from 31% to 50%, because shrinking removes persistent variance and lowers from 1.30 to 0.90. Holding the ratio at 2 and scaling both components instead takes the rate from 31% to 8%. Second, more transient inefficiency hurts: enters the denominator, so a noisier transient block contaminates the firm means and makes the persistent split harder.
Note what this quantity does and does not explain. enters it directly, so it covers the contamination channel by which longer panels help. It says nothing about , which helps through a different route – sampling error in what is effectively a deconvolution from firm-level draws – which is why the two dimensions have to be measured separately, as above.
One limit on all of that, stated because it is easy to over-read. Those elasticities were measured at against , where persistent inefficiency is the larger of the two time-invariant components. On a design where the random effect dominates instead (, , , ) the level ordering reverses: "fiml" collapses on 33% of replications and "sml" on 20%. Which estimator handles the split better is not a fact about the estimators alone – it depends on how much persistent inefficiency there is to find.
psfm therefore reports the condition rather than preventing it, in $sigh_at_bound and $sigr_at_bound and in a warning naming the surviving scale. Bounding either parameter away from zero would corrupt precisely the samples where the boundary is the answer. What the warning asks of you is to read the two persistent scales together and to treat the split between them as unidentified in that sample – their sum remains informative even where the division does not.
The two true-fixed-effects estimators, and a change of meaning in "TFE". Both "TFE" and "TFE_WMLE" fit the same model,
with an unrestricted individual effect; they differ in how they estimate it.
"TFE" is Greene's (2005, JoE/JPA) true fixed effects estimator: the ordinary composed-error likelihood with the individual intercepts estimated jointly with . This package concentrates the out at every likelihood evaluation (each is the unique root of a strictly decreasing score, solved by safeguarded Newton to machine precision), so the reported parameter vector, its standard errors and the out layout are the same 2 + K as "TFE_WMLE". It is the estimator subject to the incidental parameters problem: with fixed, is biased upward and efficiency scores correspondingly pessimistic.
"TFE_WMLE" is Chen, Schmidt and Wang's (2014, JoE) within maximum-likelihood estimator, which applies maximum likelihood to the deviations from individual means. The drop out of the likelihood entirely, so the estimator is free of the incidental parameters problem; the firm effects are recovered afterwards from the mean residual with the usual adjustment.
The two are not nested and their log-likelihoods are not on the same scale – "TFE" evaluates the likelihood of the data, "TFE_WMLE" that of the within-transformed deviations – so logLik(), AIC() and BIC() must not be used to choose between them.
Through sfa 1.1.2, model_name = "TFE" selected the Chen-Schmidt-Wang estimator. It now selects Greene's, matching standard usage; the Chen-Schmidt-Wang estimator is "TFE_WMLE". Existing scripts that pass "TFE" therefore get a different estimator than they did, and are warned once per call.
Degeneracy of the Greene likelihood. Because is unrestricted, Greene's likelihood always has a supremum on the boundary: setting makes every composed error weakly negative, , and what remains is the deterministic-frontier likelihood, which is finite and attained. On simulated panels this supremum is reached along a path that first dips below the interior local maximum and then rises past it, so an unconstrained search that escapes the interior basin returns together with meaningless standard errors; when is large and small, no interior maximum need exist at all. tfe_lambda_max bounds the search accordingly, and a fit that pins at the bound warns. This is a property of the estimator, not of the implementation, and is one of the motivations for "TFE_WMLE".
Both problems are small- phenomena. Simulations in which grows with fixed at 100 (truth ) recover 88.5, 4.8, 3.6, 3.6, 3.5 at 10, 20, 30, 40, 50: the fit is pinned at the bound at and has converged by . The degeneracy and the incidental-parameters bias dissolve together as grows, so tfe_lambda_max matters on short panels and is irrelevant on long ones. "TFE_WMLE", having no incidental parameters, is unaffected by either way.
"K1990" and "K1990modified" (Kumbhakar, 1990) extend the same error-components frontier with richer time paths for inefficiency: "K1990" uses and "K1990modified" uses , against "BC92"'s and "PL80"'s time-invariant . All four share one likelihood, differing only in . Note that "K1990"'s b and c are weakly identified – quite different pairs produce near-identical fitted time paths – so the fitted path is more interpretable than either coefficient on its own.
"PL80" (time-invariant panel inefficiency, Pitt and Lee, 1980, JoE) and "BC92" (time-varying decay, Battese and Coelli, 1992, JPA) are both natively estimated maximum-likelihood implementations of the same "error components frontier": y_it = x_it'beta + v_it - B_it*u_i (production; sign flips for cost), with v_it iid normal, u_i iid half-normal and constant across time within a firm, and B_it = 1 for "PL80" or B_it = exp(-eta*(t - Tref)) for "BC92" (Tref being the last time period in the whole panel). The closed-form log-likelihood is obtained by integrating u_i out of the joint density of (epsilon_i, u_i); it nests the ordinary cross-sectional normal-half-normal likelihood when every firm has a single observation. Verified against frontier::sfa() (matching coefficients, log-likelihood, and frontier::efficiencies()-style predicted efficiencies to numerical precision across balanced/unbalanced panels and production/cost specifications) before that dependency was removed. The time argument above supplies each observation's period for "BC92"'s decay term; if left NULL, a sequential within-individual time index is constructed from row order, which assumes data is already sorted chronologically within each individual.
"SSFE" is the classical Schmidt and Sickles (1984, JBES) fixed-effects (LSDV) estimator: a standard within/fixed-effects panel regression, with technical inefficiency read directly off the estimated firm effects (each firm's distance from the best-performing firm in the sample) rather than estimated by maximum likelihood. It requires no numerical optimization and returns no log-likelihood – logLik()/AIC()/BIC() are not defined for it (see sfareg-methods). Note this is a different model from "TFE"/"TFE_WMLE" above, despite the similar names: both of those are maximum-likelihood estimators that allow a separate stochastic noise term on top of the fixed effects, while "SSFE" attributes all within-firm deviation to inefficiency directly.
The Schmidt–Sickles family. "SSFE", "SSRE" and "SSCRE" are the within, GLS and correlated-random-effects members of one family: all three read inefficiency off a firm effect as distance from the best firm, and none assumes a distribution for . The choice among them is the standard panel trade-off, with a frontier-specific twist. Random effects is the more efficient estimator and is the only one of the three that identifies coefficients on time-invariant regressors, which the within transformation sweeps away – but it requires the effects to be uncorrelated with the regressors, and in a frontier model that assumption says a firm's inefficiency is unrelated to its input choices, which is close to saying managers do not respond to their own productivity.
"SSCRE" is the answer to that. Adding the within-firm means of the time-varying regressors (Mundlak 1978) models the correlation instead of assuming it away; the slopes on the original regressors then equal the within estimates exactly – not approximately – while time-invariant regressors remain identified. The coefficients on the added means are reported with a .mean_ prefix, and testing them jointly against zero is the Mundlak form of the Hausman test: if they are indistinguishable from zero, "SSRE" was safe. A regressor already constant within firm has a group mean identical to itself, so it contributes no Mundlak term; those are exactly the regressors random effects exists to identify, and they are left alone. The variance components, and so and the BLUP-based inefficiencies, are the Swamy–Arora components of the model without the Mundlak means, balanced or unbalanced. On a balanced panel that is what plm's Swamy–Arora step returns; on an unbalanced one that step is singular, because in the between-firm dimension each mean duplicates its regressor, and "SSCRE" obtains the identical components through plm's Wallace–Hussain step on the augmented model, checking that they agree.
Time-varying fixed effects without a distribution for u. "CSS" and "LS" sit between "SSFE", which holds inefficiency fixed over time, and the maximum-likelihood panel models, which let it move but only by assuming a distribution for and a decay path for it. Both instead let the firm effect itself vary with time and read inefficiency off it, so neither assumes any distribution and neither returns a log-likelihood.
They differ in what they are willing to spend on flexibility. Cornwell, Schmidt and Sickles (1990) give every firm its own quadratic in time, : parameters, firms may overtake one another, and is required before a firm contributes to at all (a firm observed three times is fitted exactly by its own quadratic and its residuals are identically zero – sfa warns rather than letting such firms silently dilute the estimate). Lee and Schmidt (1993) impose one common temporal pattern scaled per firm, : parameters, and the ranking of firms cannot change, only the spread. Cross-over versus no cross-over is the substantive choice between them, not a technical one.
"CSS" is fitted by residualizing and on each firm's own rather than by building dummy columns; a regressor spanned by those quadratics – a pure time trend, for instance – is a rank failure and is refused rather than reported as a small coefficient. "LS" is a rank-one factor model, so it is fitted by alternating least squares, which handles an unbalanced panel with no special-casing; fixes the scale that leaves free, with a fallback to a norm constraint when the first period's effect is near zero.
For both, inefficiency is identified only relative to the best firm in each period – – which is what makes them time-varying at all; comparing against a single overall best would collapse them back to a time-invariant ranking. Neither absorbs an intercept separately: it goes into the firm effect, as in "SSFE", so coef() returns slopes only.
Kneip, Sickles and Song (2012). "KSS" writes the firm effect as , where both the basis functions and the dimension are estimated from the data rather than assumed. That makes explicit what the other time-varying estimators here are special cases of:
| model | |
basis |
"SSFE" |
1 | fixed, constant in |
"LS" |
1 | free |
"CSS" |
3 | fixed to |
"KSS" |
estimated | estimated |
Estimation follows the three steps of the original: cross-sectional centering by period (which imposes and removes any common function of time), then smoothing of each firm's residual trajectory followed by an eigendecomposition of their empirical covariance to give , then loadings by least squares and the effects rebuilt from them – iterated with until the residual sum of squares settles. Smoothing before the eigendecomposition is what makes the estimated basis a set of smooth functions rather than arbitrary period effects, and is the whole difference between this and an unrestricted factor model.
Two tuning choices are therefore unavoidable, and both are exposed. kss_smooth defaults to generalized cross-validation. kss_L defaults to the Bai and Ng (2002) criterion – not the threshold rule of the original paper, which is stated here because it matters: is the standard criterion for this problem and the literature that motivates the estimator treats Bai and Ng (2002) and Bai (2009) as the comparable methods, but a different rule can select a different . Set kss_L explicitly to take the choice out of the criterion's hands, and inspect $kss$eigenvalues to see whether the choice was close.
The criterion needs a long enough panel, and says so. works because the residual variance flattens out once the real factors are in and the penalty then dominates. On a short panel that never happens: if is allowed up to the factors can span nearly the whole time dimension, the variance collapses toward zero, and no penalty competes. Measured at over five seeds and two true ranks, an uncapped criterion returned its maximum on all ten designs at and again at . The automatic search is therefore capped at , which recovers the true rank exactly from upward; at it is right for a one-factor structure and occasionally one too high for a two-factor one. Whenever the automatic choice lands on that cap psfm warns, because a selection at the boundary is not a free choice – read it as "at least this many", inspect the eigenvalues, and prefer a longer panel to a larger kss_L_max.
One consequence of the period centering is worth knowing before comparing models: "KSS"'s alpha_hat carries no period mean, while "CSS"'s and "LS"'s do. The two therefore differ by a per-period constant by construction and should not be correlated against each other directly. u_hat is a within-period contrast and is invariant to that shift, so it is the quantity to compare across these models – and is what the estimators are for.
mundlak)Mundlak adjustment terms (mundlak). The true random effects model assumes the firm effect is independent of the regressors. In production data it usually is not – better managers choose different input levels – and the effect is then split between the slopes and inefficiency, biasing the technology parameters and understating inefficiency. Mundlak's (1978) device models the correlation instead of assuming it away, . Substituting into the frontier leaves the same likelihood with a wider design matrix, which is why this is an argument rather than a new model_name.
Karagiannis and Kellermann (2019) widen it to , adding environmental variables – conditions outside the producer's control, such as agro-ecological conditions or the geography an airline serves. Those enter the formula as ordinary time-invariant regressors; mundlak supplies the piece a user cannot easily build by hand, the within-firm means computed against the same panel index the fit uses. Naming a subset of the regressors gives their Hausman-Taylor variant, in which only some regressors are taken to correlate with the effect.
Measured over 40 replications with , , on a design where the firm effect is correlated with by construction, and on a control where it is not:
| design | estimator | bias | RMSE | |
| effect correlated | plain | +0.0479 | 0.0570 | |
| effect correlated | mundlak |
-0.0061 | 0.0310 | |
| no correlation | plain | -0.0033 | 0.0253 | |
| no correlation | mundlak |
-0.0068 | 0.0311 |
So it removes the bias where there is one and nearly halves RMSE – and where there is no correlation it costs about 23% in RMSE for nothing. It is not free, and it is not a default.
A variable that is already constant within firm gets no mean column: its own mean is itself, so the two would be exactly collinear. Every mundlak fit also emits a collinearity warning from the starting-value regression, which is expected and harmless – in a between-individual regression and are the same column by construction. The starting values fall back to pooled OLS and the likelihood is estimated with the full requested formula.
Profiling a fitted panel likelihood. With keep_objective = TRUE the fitted object carries $objective, the same negative-summed-log-likelihood the optimizer minimised. Evaluating it away from the optimum is the only way to ask whether a parameter is weakly identified, as distinct from badly estimated:
fit <- psfm(y ~ x1 + x2, "GTRE", d, individual = "id",
estimator = "sml", keep_objective = TRUE)
ih <- which(names(coef(fit)) == "sigh")
grid <- seq(0.05, 1, by = 0.05)
ll <- vapply(grid, function(v) {
p <- fit$opt$par; p[ih] <- v; -fit$objective(p) }, numeric(1))
plot(grid, ll - max(ll), type = "l")
A flat curve means no estimator could recover the parameter from these data; a sharp one means the difficulty lies elsewhere. For GTRE's sigh the curve spans over 200 log units at and sharpens with , which is what ruled out weak identification as the explanation for its convergence behaviour.
Note on default iteration counts: psfm()'s defaults were raised in version 1.1.3, from 100/10/10 for maxit.bobyqa/maxit.psoptim/maxit.optim to 5000/100/1000, and maxit.nlminb (previously hard-coded) is now an argument. The old values were binding rather than merely economical: the K1990/K1990modified branch already carried a note that 100 bobyqa evaluations left its seven-parameter fits several log-likelihood units short of the optimum purely on the iteration cap, and GTRE_FML at N = 500, T = 10 roughly halves its root-mean-square error against known true values when the caps are lifted, at about 1.5 times the run time. They remain below sfm()'s defaults because several psfm() models (GTRE, GTRE_Z, TRE, TRE_Z) are estimated by simulated maximum likelihood over Halton draws, so each objective evaluation is far more expensive than the closed-form likelihoods used elsewhere. Lower them explicitly if a fast approximate call is wanted.
An object of class "sfareg" containing components that vary by model. All models return:
out |
A matrix with parameter estimates, standard errors, and t-values. |
opt |
A list containing the optimization results from the final optimization procedure (not returned for GTRE_SEQ1 and GTRE_SEQ2). |
total_time |
The total computation time for model estimation. |
start_v |
The starting values used in the optimization (not returned for GTRE_SEQ1 and GTRE_SEQ2). |
model_name |
The name of the panel stochastic frontier model estimated. |
formula |
The formula used in the model specification. |
coefficients |
A vector of estimated parameters. |
std.errors |
A vector of standard errors for the estimated parameters (NA if |
t.values |
A vector of t-values for the estimated parameters (NA if |
call |
The matched call. |
data |
The data used in estimation. |
Additional model-specific components:
For GTRE_FML model:
start_search |
A list recording the starting-value search: |
U |
Predicted transient (time-varying) technical efficiency for each observation. |
H |
Predicted persistent (time-invariant) technical efficiency for each individual, named by individual. |
For GTRE and GTRE_Z models:
H |
Predicted time-invariant technical efficiency for each individual. |
For GTRE_Z model:
out_opg |
OPG (outer product of gradients) standard errors and t-values along with parameters. |
out_sandwich |
sandwich standard errors and t-values along with parameters. |
For GTRE, GTRE_Z, TRE and TRE_Z models:
U |
Predicted time-varying technical efficiency for each observation. |
For TFE and TFE_WMLE models:
r_hat_m |
Estimated individual-specific fixed effects, named by individual. For |
exp_u_hat |
Predicted technical efficiency, |
For TFE model only:
u_hat |
Predicted inefficiency, |
For FD model:
u_hat |
Predicted technical efficiency in levels. |
h_hat |
Estimated z heterogeneity function values. |
exp_u_hat |
Predicted technical efficiency. |
For GTRE_SEQ1 and GTRE_SEQ2 models:
other_parms |
A matrix of additional parameters (lambda, sigma, beta_0 for SEQ1; sigma_u, sigma_v, sigma_h, sigma_r, lambda, sigma for SEQ2). |
For SSFE model:
alpha_hat |
The estimated firm-specific fixed effects (named by individual). |
u_hat |
Estimated technical inefficiency for each firm, as its distance from the best-performing firm's fixed effect. |
exp_u_hat |
Predicted technical efficiency, |
For the GTRE model:
sigh_at_bound, sigr_at_bound
|
|
For CSS and LS models:
alpha_hat |
The fitted firm effect for every observation, |
u_hat, exp_u_hat
|
Inefficiency and efficiency per observation, measured against the best firm in the same period. |
frontier_alpha |
The best firm effect in each period, i.e. the benchmark |
sigma_v, residuals, df.residual
|
The residual standard deviation, the residuals net of the firm effect, and the residual degrees of freedom ( |
theta |
|
delta, alpha_i, ls_iterations, ls_converged
|
|
For SSRE and SSCRE models:
alpha_hat |
The BLUPs of the firm effects, one per firm. Under |
theta, ercomp
|
The GLS transformation parameter and the full variance-components decomposition from |
mundlak_terms |
|
plm_fit |
The underlying |
For the KSS model:
kss |
A list: the selected dimension |
For PL80 and BC92 models:
exp_u_hat |
Predicted technical efficiency for each observation, E[exp(-B_it*u_i)|epsilon_i] (generalizes Battese and Coelli's (1988, JoE) point estimator to the panel case, pooling all of a firm's observations to predict u_i and scaling by B_it per period). |
Standard errors require optHessian = TRUE.
The GTRE_SEQ1 and GTRE_SEQ2 models use sequential estimation methods and do not return optimization objects or starting values. The SSFE, CSS, LS, SSRE, SSCRE and KSS models are likewise not maximum-likelihood based and return no optimization object, starting values, or log-likelihood. All panel models require the individual argument to identify panel units.
David H. Bernstein
Fillipini and Greene (2016, JPA); Colombi, Kumbhakar, Martini and Vittadini (2014, JPA) on the closed-skew-normal four-component likelihood, with Colombi (2010) and Colombi, Martini and Vittadini (2011) on the two-step estimator used to seed it; Greene (2005, JoE and JPA); Wang and Ho (2010, JoE); Chen, Schmidt and Wang (2014, JoE); Kumbhakar (1990, JoE); Pitt and Lee (1980, JoE); Battese and Coelli (1992, JPA); Battese and Coelli (1988, JoE); Jondrow, Lovell, Materov and Schmidt (1982, JoE); Schmidt and Sickles (1984, JBES)
sfm for cross-sectional models, psfm_bootstrap for bootstrap standard errors, data_gen_p for simulating panel data with known true parameters, and sfareg-methods for extracting results.
library(sfa) ## A deliberately small panel with few simulation draws, so the examples ## build quickly. Do not read the estimates as a serious fit. TRE_Z becomes ## unstable below about 60 firms, so this is close to the floor. data_trial <- data_gen_p(t=6, N=60, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) ## rand.psoptim is not optional with PSopt = TRUE: the particle swarm draws ## from the session RNG, so results otherwise change between builds. max_tre_z <- psfm(formula = y_tre_z ~ x1 +x2| z_gtre, model_name = "TRE_Z", data = data_trial, individual = "name", halton_num = 50, rand.psoptim = 7L, PSopt = TRUE) ## Time-varying fixed effects, no distribution assumed for u. CSS gives each ## firm its own quadratic in time; LS gives all firms one common pattern. css_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "CSS", data = data_trial, individual = "name", time = "year") ls_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "LS", data = data_trial, individual = "name", time = "year") ls_fit$delta # the common temporal pattern, normalized to delta[1] = 1 ## The Schmidt-Sickles family: within, GLS, and correlated random effects. ## SSCRE's slopes equal SSFE's within slopes exactly (Mundlak 1978); the ## .mean_ coefficients are the Mundlak form of the Hausman test. ssre_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "SSRE", data = data_trial, individual = "name", time = "year") sscre_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "SSCRE", data = data_trial, individual = "name", time = "year") ## Kneip, Sickles and Song (2012): the temporal basis and its dimension are ## both estimated. Inspect what the criterion chose before trusting it. kss_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "KSS", data = data_trial, individual = "name", time = "year") kss_fit$kss$L # selected dimension kss_fit$kss$eigenvalues # was the choice close?library(sfa) ## A deliberately small panel with few simulation draws, so the examples ## build quickly. Do not read the estimates as a serious fit. TRE_Z becomes ## unstable below about 60 firms, so this is close to the floor. data_trial <- data_gen_p(t=6, N=60, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = .2, sig_h = .4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) ## rand.psoptim is not optional with PSopt = TRUE: the particle swarm draws ## from the session RNG, so results otherwise change between builds. max_tre_z <- psfm(formula = y_tre_z ~ x1 +x2| z_gtre, model_name = "TRE_Z", data = data_trial, individual = "name", halton_num = 50, rand.psoptim = 7L, PSopt = TRUE) ## Time-varying fixed effects, no distribution assumed for u. CSS gives each ## firm its own quadratic in time; LS gives all firms one common pattern. css_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "CSS", data = data_trial, individual = "name", time = "year") ls_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "LS", data = data_trial, individual = "name", time = "year") ls_fit$delta # the common temporal pattern, normalized to delta[1] = 1 ## The Schmidt-Sickles family: within, GLS, and correlated random effects. ## SSCRE's slopes equal SSFE's within slopes exactly (Mundlak 1978); the ## .mean_ coefficients are the Mundlak form of the Hausman test. ssre_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "SSRE", data = data_trial, individual = "name", time = "year") sscre_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "SSCRE", data = data_trial, individual = "name", time = "year") ## Kneip, Sickles and Song (2012): the temporal basis and its dimension are ## both estimated. Inspect what the criterion chose before trusting it. kss_fit <- psfm(formula = y_tre ~ x1 + x2, model_name = "KSS", data = data_trial, individual = "name", time = "year") kss_fit$kss$L # selected dimension kss_fit$kss$eigenvalues # was the choice close?
psfm() Panel Stochastic Frontier ModelsParametric bootstrap standard errors for a panel frontier fitted by
psfm. For each of BOOT replications a new response
is simulated from the fitted parameters, with a data-generating process
specific to the model, and the model is refitted in parallel. Bootstrap
standard errors and t-values are returned for every parameter in
psfm_object$out.
Supported models: "GTRE_Z", "TRE_Z", "GTRE",
"GTRE_FML", "TRE", "TFE", "TFE_WMLE" and
"FD". See ‘Supported models’ in Details for why the others are
refused.
psfm_bootstrap(psfm_object, numCores, BOOT, individual, h_type = c("auto", "none", "scalar", "parametric"), maxit.psoptim = 1000, seed_offset = 0, write_back = TRUE, pkgs = c("sfa", "Formula", "truncnorm"), inefdec, rand.gtre = NULL, rand.psoptim = NULL, maxit.bobyqa = 1, maxit.optim = 1)psfm_bootstrap(psfm_object, numCores, BOOT, individual, h_type = c("auto", "none", "scalar", "parametric"), maxit.psoptim = 1000, seed_offset = 0, write_back = TRUE, pkgs = c("sfa", "Formula", "truncnorm"), inefdec, rand.gtre = NULL, rand.psoptim = NULL, maxit.bobyqa = 1, maxit.optim = 1)
psfm_object |
A fitted model object of class |
numCores |
Integer. Number of cores for the parallel cluster created
internally via |
BOOT |
Integer. Number of bootstrap replications (e.g. 199, 399, 999). |
individual |
Character. Name of the individual/panel-ID column in
|
h_type |
One of
|
maxit.psoptim |
Integer. Passed through to |
seed_offset |
Whole number selecting a block of seeds: replication
|
write_back |
Logical. If |
pkgs |
Character vector of package names to load on each cluster
worker via |
inefdec |
Logical: |
rand.gtre |
Passed through to |
rand.psoptim |
Passed through to |
maxit.bobyqa |
Integer. Passed through to |
maxit.optim |
Integer. Passed through to |
Any other model_name is refused with an error. The moment-based,
sequential and LSDV estimators ("GTRE_SEQ1", "GTRE_SEQ2",
"SSFE", "SSRE", "SSCRE", "CSS", "LS",
"KSS") are not maximum likelihood, so there is no fitted likelihood to
resample from in the same sense. The time-pattern models ("PL80",
"PL80_MVTN", "BC92", "K1990", "K1990modified") do
not return the $U/$H structure this function relies on.
The supported models fall into three families, each with its own
data-generating process and $out row layout. The header comment of
R/psfm_bootstrap.R gives the full derivation, including how each was
cross-checked against that model's own likelihood in psfm.R.
"randeff" family (GTRE_Z, TRE_Z, GTRE,
TRE)y* = X %*% beta_hat + v + u + r + h (sign per
inefdec), with v ~ N(0, sigma_v^2), u half-normal
(covariate-driven sigma for GTRE_Z/TRE_Z, a single
homoskedastic scalar for bare GTRE/TRE, which have no
z pipe at all), r ~ N(0, sigma_r^2) (one draw per
individual, repeated over time), and h per h_type
(GTRE/GTRE_Z only). $out row layout: sigv,
sigr, [x-block], [z-block], [h-block] for GTRE_Z/TRE_Z;
lambda, sigma, sigr, [sigh], [x-block] for bare
GTRE/TRE (note the different reparameterization – see
psfm's own model-family sigma-link inconsistency
note).
"tfe" family (TFE, TFE_WMLE)Fixed-effects model: the individual
effects are held FIXED at psfm_object$r_hat_m's original point
estimates across every replication (not redrawn – redrawing them would
contradict treating them as fixed rather than random), and only
v ~ N(0, sigma_v^2) and u ~ |N(0, sigma_u^2)| (both derived
from the fitted lambda/sig, or gamma/sig if
the original fit used psfm(..., gamma = TRUE)) are redrawn.
$out row layout: lambda (or gamma), sig, [x-block] (no
intercept row – the within-transformation removes it).
"fd" family (FD)Wang and Ho (2010) first-difference
estimator: a single time-invariant u_i ~ TruncatedNormal(mean =
mu, sd = sqrt(sig_u2), lower = 0) draw per individual (via
rtruncnorm), scaled by a deterministic
time-varying factor h_it = exp(z_it %*% delta), plus
v_it ~ N(0, sig_v2). $out row layout: sig_u2, sig_v2,
mu, [x-block], [z-block] (no intercept row in either the x- or z-block).
psfm_bootstrap validates that nrow(psfm_object$out) matches the
expected layout for the resolved model/h_type before running any
bootstrap replications, and stops with an informative error if it does not.
For GTRE/GTRE_Z it similarly validates up front that
length(psfm_object$H) equals the number of unique individuals implied
by the individual column ($H is one value per individual, not
one per observation) – TRE/TRE_Z/TFE/TFE_WMLE/FD do not
return $H at all, so boot_eff_h is simply NULL for those
five models (not an error).
The model is re-estimated on each simulated dataset via sfa::psfm(),
executed in parallel across replications using
parLapply (or pblapply for a
live progress bar, when the pbapply package is installed). Whether the
psoptim (particle-swarm) optimizer stage is used on each refit is
model-dependent: it is enabled for the "randeff" family (matching this
function's original behavior, tested clean), but disabled for "TFE" and "TFE_WMLE"
– testing found psoptim's wide exploration made TFE
repeatedly collapse onto a degenerate variance-boundary solution (reproduced
even refitting the SAME real data across different psoptim seeds), a
problem that disappeared entirely once that stage was skipped (matching
psfm()'s own default of PSopt = FALSE). As a defense-in-depth
safety net across every family, any replication whose refit still lands on a
degenerate variance-boundary mode (a scale parameter pinned near its
numerical floor – the same well-understood boundary-mode phenomenon
documented for ttsfm()'s TTNE/TTHN models) is treated
as a per-replication FAILURE rather than silently corrupting the bootstrap
standard error with a meaningless draw.
Re-estimation failures (optimizer errors, or the degenerate-mode check just
described) are caught per replication, recorded with their message, and set
to NA in the output matrices rather than stopping the bootstrap; a
single warning at the end lists every failed replication index
together with its message. Bootstrap standard errors are computed with
na.rm = TRUE, so they remain well-defined even when some replications
failed.
Requires the parallel and Formula packages (and truncnorm
when bootstrapping an "FD" fit), and requires sfa (and any other
packages named in pkgs) to be installed and loadable on each cluster
worker.
A list with components:
boot_par |
A |
boot_eff |
A |
boot_eff_h |
Only non- |
se |
Named numeric vector of bootstrap standard errors for every
parameter in |
tval |
Named numeric vector of bootstrap t-values
( |
failures |
Integer vector of replication indices |
model |
Only present when |
David H. Bernstein and Christopher F. Parmeter
psfm, parLapply,
pblapply, Formula
## Not run: ## Not run: a meaningful bootstrap needs BOOT in the hundreds, each ## replication refitting the model on a PSOCK cluster, so this takes ## minutes rather than seconds. data_trial <- data_gen_p(t = 5, N = 50, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = 0.2, sig_h = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) GTRE_ZZ <- psfm(y_gtre_zz ~ x1 + x2 | z_gtre | zp_gtre, model_name = "GTRE_Z", data = data_trial, individual = "name") boot_results <- psfm_bootstrap( psfm_object = GTRE_ZZ, numCores = 2, BOOT = 199, individual = "name", h_type = "auto" ) ## inefdec is left out on purpose: it is recovered from the fit, which used ## psfm()'s default inefdec = TRUE because y_gtre_zz is a production frontier. ## Bootstrap standard errors / t-values for every model parameter boot_results$se boot_results$tval ## Bootstrap draws of time-varying (U) and time-invariant (H) ## efficiency/inefficiency scores, one row per replication boot_results$boot_eff boot_results$boot_eff_h ## Fitted object with bootstrap SEs/t-values written into $out boot_results$model$out ## End(Not run)## Not run: ## Not run: a meaningful bootstrap needs BOOT in the hundreds, each ## replication refitting the model on a PSOCK cluster, so this takes ## minutes rather than seconds. data_trial <- data_gen_p(t = 5, N = 50, rand = 100, sig_u = 1, sig_v = 0.3, sig_r = 0.2, sig_h = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5) GTRE_ZZ <- psfm(y_gtre_zz ~ x1 + x2 | z_gtre | zp_gtre, model_name = "GTRE_Z", data = data_trial, individual = "name") boot_results <- psfm_bootstrap( psfm_object = GTRE_ZZ, numCores = 2, BOOT = 199, individual = "name", h_type = "auto" ) ## inefdec is left out on purpose: it is recovered from the fit, which used ## psfm()'s default inefdec = TRUE because y_gtre_zz is a production frontier. ## Bootstrap standard errors / t-values for every model parameter boot_results$se boot_results$tval ## Bootstrap draws of time-varying (U) and time-invariant (H) ## efficiency/inefficiency scores, one row per replication boot_results$boot_eff boot_results$boot_eff_h ## Fitted object with bootstrap SEs/t-values written into $out boot_results$model$out ## End(Not run)
Fits one of two models in which the frontier and a binary choice are
estimated together. model_name picks the model:
model_name |
model | what drives the choice |
"greene" (default) |
Greene (2010): one frontier, seen only for units that select into the sample | the noise |
"kts" |
Kumbhakar, Tsionas and Sipilainen (2009): a frontier and two scales per technology | inefficiency
|
The two are different models, not variants of one, and neither nests the
other. "greene" is a two-step estimator: a probit on the full sample,
then maximum simulated likelihood for the frontier on the selected rows.
"kts" is single-step maximum likelihood over the whole sample, with the
integral over done by Gauss–Legendre quadrature.
selsfm(selection, frontier, data, model_name = c("greene", "kts"), n_nodes = 64, Nsim = "auto", sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, seed = NULL, inefdec = TRUE, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)selsfm(selection, frontier, data, model_name = c("greene", "kts"), n_nodes = 64, Nsim = "auto", sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, seed = NULL, inefdec = TRUE, maxit.bobyqa = 10000, maxit.psoptim = 1000, maxit.optim = 1000, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, Method = "L-BFGS-B", verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)
selection |
A two-sided formula for the binary equation, |
frontier |
A two-sided formula for the frontier, |
model_name |
|
n_nodes |
|
data |
A |
Nsim |
|
sim_type |
Low-discrepancy sequence for the draws, passed to the same generator the other simulated-ML models use. |
antithetics |
Mirror the draws to halve the number of independent points. |
seed |
Optional seed for the draws. The RNG state is saved and restored, so a fit does not disturb the caller's stream. |
inefdec |
|
maxit.bobyqa, maxit.psoptim, maxit.optim
|
Iteration caps for the three optimizer stages. |
start_val |
For |
PSopt |
Run the particle-swarm stage between BOBYQA and |
optHessian |
Compute the Hessian, and with it the standard errors. |
Method |
Method passed to |
verbose |
Report optimizer progress. |
keep_objective |
Logical. Retain the second-stage likelihood closure on the fitted object as |
rand.psoptim |
Optional seed for the particle-swarm stage. |
The model is Greene's Equation (10):
Selection operates through the noise , not through the
inefficiency . That is the substantive assumption separating this model
from Kumbhakar, Tsionas and Sipilainen (2009), where the selection disturbance
is correlated with instead; the two answer different questions and are
not nested.
Because cannot be integrated out in closed form, the log-likelihood is
simulated (Greene's Equation (15)): for each selected observation draws
of are taken and
with . Everything is evaluated
in logs and accumulated by log-sum-exp, since the summand underflows in the
tails well before the total does.
A note on the published formula. Equations (11) and (13)–(15) of Greene
(2010) write in the denominator of the
argument. It must be : the term being standardized is
, which is the noise,
and the conditional distribution of given for a bivariate normal
with and correlation is
. This implementation uses .
Standard errors are conditional on the first stage. Following the paper's
two-step scheme, is taken as given in the frontier likelihood. The
reported standard errors come from the second-stage Hessian and therefore do
not include the sampling variability of ; Greene applies
the Murphy–Topel correction at this point, which is not implemented here.
Treat them as a lower bound, and bootstrap over both stages when the precision
matters.
For technology ,
is the parameter the model exists for: it is the dual role
of inefficiency, and means the less efficient are more likely
to choose technology 1. Neither two-step order works – the choice equation
cannot be a probit because is unobserved, and fitting the frontiers
first ignores the endogeneity of the choice – which is why the paper estimates
in one step.
Because 's distribution depends on while 's probability
depends on , the marginal density of is a mixture of the
two regimes' half-normals, with weight solving the paper's
equation (9). The observation's density is then the same mixture of two
one-dimensional integrals, its equation (14).
Recovery. On data simulated from this exact model with
, the frontier coefficients, the four scales and
all recover, and improve with the sample:
| parameter | n = 1500 | n = 6000 | truth |
|
1.361 | 0.922 | 0.900 |
|
0.837 | 0.843 | 0.800 |
|
0.632 | 0.640 | 0.600 |
| choice intercept | -0.511 | -0.239 | -0.200 |
is the parameter that needs the most data, as one would expect of
a coefficient identified only through the coupling between the two equations,
but it is properly identified: holding the rest at the truth, the mean
log-likelihood falls monotonically away from in both
directions, by about 0.069 per observation across .
An object of class "sfareg". As elsewhere in the package out is a
p x 3 matrix, one row per parameter, with columns par,
st_err and t-val. Both models return jlms,
efficiency (exp(-jlms)), selection_formula and
nobs.
For "greene" (model_name "SEL" on the fit), the
parameters are sigma_u, sigma_v, rho, then the frontier
coefficients, and nobs is the number of selected rows, the sample the
second-stage likelihood sums over. Also returned:
alpha |
First-stage probit coefficients. |
probit |
The fitted first-stage object. |
jlms |
|
n_selected, n_total
|
Selected and total observation counts. |
Nsim, sim_type
|
The draw count and sequence actually used. |
For "kts" (model_name "KTS" on the fit), the parameters
are the technology-0 frontier (t0.), the technology-1 frontier
(t1.), sigma_v0, sigma_v1, sigma_u0,
sigma_u1, the choice equation (choice.), then delta.
nobs counts every row used, n_selected the rows with
technology 1, and n_nodes the quadrature nodes.
Kumbhakar, S.C., Tsionas, E.G. and Sipilainen, T. (2009). Joint estimation of technology choice and technical efficiency: an application to organic and conventional dairy farming. Journal of Productivity Analysis, 31(3), 151–161.
Greene, W. (2010). A stochastic frontier model with correction for sample selection. Journal of Productivity Analysis, 34(1), 15–24.
Heckman, J. (1979). Sample selection bias as a specification error. Econometrica, 47(1), 153–161.
sfm for the frontier without a selection correction.
set.seed(1) n <- 800 z1 <- rnorm(n); x1 <- rnorm(n) w <- rnorm(n) d <- as.numeric(0.2 + 0.8 * z1 + w > 0) v <- 0.5 * (0.6 * w + sqrt(1 - 0.6^2) * rnorm(n)) y <- 0.5 + 0.8 * x1 + v - abs(rnorm(n)) y[d == 0] <- NA dat <- data.frame(y = y, x1 = x1, z1 = z1, d = d) fit <- selsfm(selection = d ~ z1, frontier = y ~ x1, data = dat, Nsim = 100) fit$outset.seed(1) n <- 800 z1 <- rnorm(n); x1 <- rnorm(n) w <- rnorm(n) d <- as.numeric(0.2 + 0.8 * z1 + w > 0) v <- 0.5 * (0.6 * w + sqrt(1 - 0.6^2) * rnorm(n)) y <- 0.5 + 0.8 * x1 + v - abs(rnorm(n)) y[d == 0] <- NA dat <- data.frame(y = y, x1 = x1, z1 = z1, d = d) fit <- selsfm(selection = d ~ z1, frontier = y ~ x1, data = dat, Nsim = 100) fit$out
Reports how the optimizer actually finished and how well the likelihood identifies the parameters: the convergence code and message, the eigenvalue spectrum and conditioning of the Hessian, the implied parameter correlations, and – where the objective was retained – the gradient at the reported optimum and a likelihood slice through each parameter.
sfa_diagnostics(object, ...) ## S3 method for class 'sfareg' plot(x, which = 1:4, n_grid = 41, span = 0.25, ...) ## S3 method for class 'sfadiag' print(x, ...)sfa_diagnostics(object, ...) ## S3 method for class 'sfareg' plot(x, which = 1:4, n_grid = 41, span = 0.25, ...) ## S3 method for class 'sfadiag' print(x, ...)
object, x
|
An object of class |
which |
Which panels to draw: |
n_grid |
Number of points in each likelihood slice. Defaults to |
span |
Half-width of each slice, as a fraction of |
... |
Currently unused. |
Every stochastic frontier fit in this package runs a staged minimizer and stores optim()'s convergence code, message, evaluation counts and numerically differentiated Hessian. None of that was reported: a fit that stopped on the iteration cap printed exactly like a converged one. sfa_diagnostics() is the reporting side of that.
What the Hessian tells you. optim() returns the Hessian of the negative log-likelihood, so at a genuine interior optimum it is positive definite. A non-positive eigenvalue means the reported point is not a minimum in that direction. A very small one means the likelihood is nearly flat along it, which is what weak identification looks like numerically – $hessian$flat_direction names the parameters that load on that flattest axis, and they are usually the pair the data cannot separate.
Slices, not profiles. Panel 3 varies one parameter and holds the others at their estimates. A true profile would re-optimize the remaining parameters at every grid point, at the cost of one optimization per point. The slice is the right object for reading curvature at the optimum and for the simplest check there is: the estimate should sit at the peak of its own slice. If it does not, the optimizer stopped early.
Retaining the objective. The gradient and the slices need the likelihood itself, which an ordinary fit does not keep. Pass keep_objective = TRUE to sfm to store it. This is off by default because a closure carries its enclosing environment, so a fit saved with one serializes the estimation data alongside the results – on a 200-observation example the fitted object grows from roughly 38 KB to 1.7 MB.
The convergence code alone is not diagnostic, and the flags do not treat it as though it were. Across NHN, NE and NTN at , and , code 52 (“ABNORMAL_TERMINATION_IN_LNSRCH”) turns up routinely alongside a maximum relative gradient of about and a positive definite Hessian: the staged minimizer had already found the optimum and the final L-BFGS-B stage simply could not take a step away from it. The same code on NTN at came with a relative gradient of and an indefinite Hessian, which is a genuine failure. What separates the two is the gradient and the Hessian, so the verdict is built from those and the code is reported as context:
a line-search code (51/52) together with a small relative gradient and a positive definite Hessian. Reported as noise.
a line-search code with no retained objective, so there is no gradient to settle it. Reported as an open question rather than resolved in the fit's favour – refit with keep_objective = TRUE.
anything else, including code 1 (the iteration limit), which is never excused: it means the search was still moving when it ran out of budget, whatever the curvature at the stopping point.
Flags are raised for a non-zero convergence code, a Hessian that is not positive definite or is worse than conditioned, any parameter pair correlated beyond , and a large relative gradient. They are phrased as statements about what the fit means, not as generic warnings.
sfa_diagnostics() returns an object of class "sfadiag": a list with model_name, call, pnames (the REPORTED parameter names), enames (the ESTIMATION-scale names, which differ where a fit estimates parameters it does not report, as ivsfm("IVLIML") does), estimates, convergence (code, meaning, message, counts, logLik, objective, objective_type), hessian (eigenvalues, condition, pos_def, flat_direction), correlation (matrix, worst_pair, worst_value); convergence additionally carries benign_nonzero and unverified_nonzero, gradient (gradient, max_abs, relative, max_rel; NULL without a retained objective), flags, and has_objective.
plot.sfareg() is called for its side effect and returns the "sfadiag" object invisibly. print.sfadiag() returns its argument invisibly.
The Hessian, the gradient and the likelihood slices are indexed by the vector
the optimizer worked on, so they are named with enames. The parameter
correlation is built from vcov, which is already on the
reported scale, so it is named with pnames. Where the two differ,
print() says so. convergence$logLik is the log-likelihood and
is NA where none is defined (the robust divergence estimators);
convergence$objective is the value the optimizer actually achieved,
whatever objective that was, and objective_type names it.
David H. Bernstein
sfm, sfareg-methods, summary.sfareg
library(sfa) d <- data_gen_cs(N = 300, rand = 3, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) ## Retaining the objective enables the gradient and the likelihood slices f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) sfa_diagnostics(f) ## Four-panel diagnostic display plot(f) ## Works without the objective too, on the Hessian alone g <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) sfa_diagnostics(g)$hessian$conditionlibrary(sfa) d <- data_gen_cs(N = 300, rand = 3, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.5) ## Retaining the objective enables the gradient and the likelihood slices f <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) sfa_diagnostics(f) ## Four-panel diagnostic display plot(f) ## Works without the objective too, on the Hessian alone g <- sfm(y_pcs ~ x1 + x2, model_name = "NHN", data = d) sfa_diagnostics(g)$hessian$condition
A guide to the options this package offers when the OLS residuals of a production frontier are skewed positively – the “wrong” way – and the estimated inefficiency collapses to zero. This page is documentation only; each option is a function or an argument documented in its own right.
The problem. Waldman (1982) showed that for the Normal-Half-Normal frontier, if the third central moment of the OLS residuals is positive then the OLS fit is a local maximum of the likelihood at . The fit then reports an industry in which every firm is fully efficient. Horrace and Wright (2020) extend the result to the truncated-normal and doubly-truncated-normal cases. The same thing happens to the moment estimator for a different reason: has no real solution with when . Olson, Schmidt and Waldman (1980) call these the Type I failure (no admissible ) and the Type II failure (no admissible ).
It is not a data problem. Simar and Wilson (2009) make the point that has to be kept in view: observing wrong skew is entirely consistent with the model being correctly specified. It happens routinely when is small relative to , and the smaller the sample the more often. So respecifying the model in response to it is, in their phrase, a solution in search of a problem – unless there is an independent reason to prefer the new specification.
What the package offers. They divide into three kinds.
Keep the model, change the estimator.
sfm(estimator = "acols")Parmeter and Zhao (2023). Corrected OLS with the first absolute moment in place of the third central moment. The third moment is the only part of the inversion that can fail, so removing it removes the failure: on the authors' own design at , , Type I failures fall from 48% to 15%.
sfm(estimator = "cmle")Zhao and Parmeter (2022). Maximum likelihood subject to two moment conditions the assumed density implies. The Waldman stationary point does not satisfy them, so the search is kept away from it. On wrong-skew samples the proportion of fits returning falls from about 96% to about 3%.
Keep the model, change the question.
skewness_test, symmetry_test
Ask first whether the residual skew is significantly wrong at all, rather than reading its sign off a point estimate. Often it is not, and nothing further is needed.
skewness_decompBonanno, De Giovanni and Domma (2017). Splits the composed third moment into the parts due to the inefficiency, to the noise, and to the dependence between them, and reports Zenga's median-based measure alongside. Under the textbook specification two of the three are identically zero, which is the whole reason a positive residual skew has nowhere to go.
moment_rangeWhat skewness a given pair of distributions can produce at all. A residual skew outside the achievable range is evidence about the distributions, not about the firms.
Change the model.
esfmHafner, Manner and Simar (2018). Widens the distribution of by a finite-sample correction that admits either sign of skew and converges to the assumed density as grows.
copsfmBonanno, De Giovanni and Domma (2017), generalizing Smith (2008). A generalized-logistic noise, which can itself be skewed either way, and a copula linking and . Either ingredient can account for a positive composed skew without setting to zero.
sfm(model_name = "NB")Carree (2002). A binomial inefficiency, the one distribution here that can be skewed either way: means most firms carry considerable inefficiency and only a few sit near the frontier. Same sample moment, opposite economics. Corrected OLS only.
sfm(model_name = "NGB2")A generalized beta of the second kind inefficiency, whose shape parameters accommodate a much wider range of third and fourth moments than the half-normal.
Choosing. If the skew is not significant, stop. If it is, prefer an estimator change to a model change unless there is a substantive reason for the new distribution, since a respecified model answers a different question. "cmle" and "acols" are the cheapest options and leave the model alone. Among the respecifications, copsfm is the most informative when the mechanism matters, because skewness_decomp then says which ingredient is doing the work.
Bonanno, G., De Giovanni, D. and Domma, F. (2017). The “wrong skewness” problem: a re-specification of stochastic frontiers. Journal of Productivity Analysis 47(1), 49-64. doi:10.1007/s11123-017-0492-8
Carree, M. A. (2002). Technological inefficiency and the skewness of the error component in stochastic frontier analysis. Economics Letters 77(1), 101-107. doi:10.1016/S0165-1765(02)00119-2
Hafner, C. M., Manner, H. and Simar, L. (2018). The “wrong skewness” problem in stochastic frontier models: A new approach. Econometric Reviews 37(4), 380-400. doi:10.1080/07474938.2016.1140284
Olson, J. A., Schmidt, P. and Waldman, D. M. (1980). A Monte Carlo study of estimators of stochastic frontier production functions. Journal of Econometrics 13(1), 67-82. doi:10.1016/0304-4076(80)90043-3
Parmeter, C. F. and Zhao, S. (2023). An alternative corrected ordinary least squares estimator for the stochastic frontier model. Empirical Economics 64(6), 2831-2857. doi:10.1007/s00181-023-02401-1
Simar, L. and Wilson, P. W. (2009). Inferences from cross-sectional, stochastic frontier models. Econometric Reviews 29(1), 62-98. doi:10.1080/07474930903324523
Waldman, D. M. (1982). A stationary point for the stochastic frontier likelihood. Journal of Econometrics 18(2), 275-279. doi:10.1016/0304-4076(82)90041-0
Zhao, S. and Parmeter, C. F. (2022). The “wrong skewness” problem: Moment constrained maximum likelihood estimation of the stochastic frontier model. Economics Letters 221, 110901. doi:10.1016/j.econlet.2022.110901
sfm, copsfm, esfm, skewness_decomp, skewness_test, moment_range
Standard extractor methods for objects of class "sfareg", which
sfm, psfm, zsfm,
ttsfm, lcsfm, copsfm,
ivsfm, selsfm and esfm return.
They let a fit be used with the usual R idioms – coef(fit),
vcov(fit), logLik(fit), AIC(fit), BIC(fit) –
instead of reaching into the returned list by hand.
| method | returns |
coef |
the reported parameters, object$coefficients |
vcov |
their covariance: inverse Hessian (default), BHHH, sandwich or clustered |
logLik |
the maximized log-likelihood, with "df" and "nobs" for AIC() and BIC() |
nobs |
the number of observations the fit used |
estfun, bread |
scores and bread for the sandwich package (need keep_objective = TRUE)
|
AIC() and BIC() are not defined for "sfareg"; the
stats defaults work through logLik and nobs.
## S3 method for class 'sfareg' coef(object, ...) ## S3 method for class 'sfareg' vcov(object, type = c("hessian", "bhhh", "sandwich", "clustered"), cluster = NULL, ...) ## S3 method for class 'sfareg' logLik(object, ...) ## S3 method for class 'sfareg' nobs(object, ...)## S3 method for class 'sfareg' coef(object, ...) ## S3 method for class 'sfareg' vcov(object, type = c("hessian", "bhhh", "sandwich", "clustered"), cluster = NULL, ...) ## S3 method for class 'sfareg' logLik(object, ...) ## S3 method for class 'sfareg' nobs(object, ...)
type |
For |
cluster |
For |
object |
An object of class |
... |
Currently unused; included for S3 method consistency. |
coef.sfareg returns the named vector of reported parameters,
object$coefficients.
vcov.sfareg with the default type = "hessian" inverts the
Hessian stored on the fit (object$opt$hessian). Where the Hessian's
diagonal is positive it is first scaled to unit diagonal, so that parameters
on very different scales do not make an invertible matrix look singular.
Where the fit estimates on a different scale from the one it reports (see
‘The scale the scores live on’ below), the inverse is then mapped onto
the reported scale. If the inversion or the mapping fails it warns and falls
back to a diagonal matrix built from the reported standard errors
(object$std.errors), which it also returns, without a warning, for a
fit that stores no Hessian. With neither it returns a matrix of NAs
with a warning. The other three types are described below.
logLik.sfareg returns the maximized log-likelihood as a
"logLik" object whose "df" and "nobs" attributes make
AIC() and BIC() work; see ‘Two quantities that are easy
to confuse’ for what "df" counts.
nobs.sfareg returns the number of observations the fit used,
not the rows supplied, so BIC() divides by the right . It takes
the first of: the count stored on the fit (object$nobs); the rows of
stored data (object$data); the length of a per-observation vector the
fit stores (such as exp_u_hat); and, failing all of those, the rows of
the call's data argument re-evaluated in the caller's frames, accepted
only if it matches what the fit was made from. It returns NA if none of
these is available.
vcov(type = "sandwich") returns the heteroskedasticity-robust sandwich and vcov(type = "clustered") its cluster-robust form. Both are formed on the ESTIMATION scale – the scale the scores and the Hessian live on – and only then mapped to the scale coef reports. That mapping is why they are not left to sandwich::sandwich(): that function composes bread and meat itself and has no way to know the two scales differ. For ttsfm, copsfm, ivsfm and psfm's PL80 family they do differ, and for ivsfm("IVLIML") they differ in dimension as well, since it maximises over 11 parameters and reports 6.
sandwich::sandwich() and sandwich::vcovCL() remain available and now return an estimation-scale covariance, which is internally consistent but not on the scale coef reports. Where a model reports what it estimates – sfm, zsfm, lcsfm, psfm's TRE/GTRE family – the two routes agree exactly, and the cluster adjustment matches vcovCL's default of .
vcov(fit, type = "bhhh") returns the inverse of the outer product of the per-observation scores instead of the inverse Hessian. Its value is that it needs no Hessian at all: it is defined whenever the scores are, which is exactly the case where the default path fails. When the Hessian is singular vcov otherwise falls back to a diagonal approximation built from the reported standard errors, discarding every covariance; BHHH keeps them. It requires keep_objective = TRUE for the same reason estfun does, and is returned on the same scale coef reports (see below). It is not the default, because it is only as good as the information-matrix equality and so disagrees with the Hessian under misspecification – which is useful when the two are compared on purpose and misleading when they are not. At a parameter bound the scores in the constrained directions are not defined: the likelihood returns a stability penalty rather than a value, so estfun falls back to a one-sided difference on the admissible side, uses zero where neither side is admissible, and warns. BHHH errors built from such a fit understate the uncertainty in those directions.
vcov returns the inverse Hessian, which is valid only if the likelihood is correctly specified and the observations are independent – neither being safe to assume in applied frontier work. Registering bread and estfun makes the sandwich package operate on these fits, so sandwich::sandwich() gives Huber–White errors and sandwich::vcovCL() gives clustered ones, and either can be handed to lmtest::coeftest():
fit <- sfm(y ~ x1 + x2, model_name = "NHN", data = d,
keep_objective = TRUE)
lmtest::coeftest(fit, vcov. = sandwich::vcovCL(fit, cluster = d$firm))
Two requirements. The fit must have been made with keep_objective = TRUE, since the scores are differenced from the retained likelihood; and the robust divergence estimators are excluded, because they do not maximise a log-likelihood and so have no score. For psfm panel fits the scores are per firm, so BHHH and sandwich standard errors are clustered by firm, and bread() scales by the number of firms rather than by nobs(). sandwich::vcovHC() does not work and is not expected to: its finite-sample corrections are built from hat values, which are not defined for a nonlinear likelihood whose parameter vector includes variance components.
keep_objective = TRUE is available on every entry point whose likelihood is separable into per-observation contributions: sfm, zsfm, lcsfm, ttsfm ("TTNE" and "TTHN"), copsfm, ivsfm ("IVLIML" and "IVCF"), selsfm, and psfm for "GTRE" (estimator = "sml"), "TRE", "GTRE_Z", "TRE_Z", "GTRE_FML", "TFE_WMLE", "FD", "PL80", "PL80_MVTN", "BC92", "K1990" and "K1990modified", whose contributions are per firm. All but sfm and psfm's four simulated-ML models are new in 1.2.1. The estimators that maximise no log-likelihood have no score and say so rather than returning an unusable fit: ttsfm("TTNLS") is nonlinear least squares, ivsfm("C2SLS") is 2SLS with a corrected intercept, psfm's "TFE", "SSFE", "SSRE", "SSCRE", "CSS", "LS", "KSS", "GTRE_SEQ1" and "GTRE_SEQ2" are moment-based, FE/LSDV or sequential, and npsfm is nonparametric.
estfun differences the retained likelihood at fit$opt$par, which is not always what coef reports: ttsfm, copsfm and ivsfm estimate log sigmas and report sigmas, ivsfm reports , and ivsfm("IVLIML") additionally estimates reduced-form parameters it never reports. vcov therefore carries the fit's own Jacobian from the estimation scale onto the reported one – the same delta method the reported standard errors already use – and marginalises any parameters that are estimated but not reported by taking the corresponding block of the inverse, not the inverse of the block. estfun itself returns the scores on the estimation scale, with that map attached as attributes, so its column count can exceed length(coef(fit)). From 1.2.1 this correction is applied on the Hessian path too: before it, sqrt(diag(vcov(f))) on a copsfm fit disagreed with the standard errors the same fit printed by a factor of , and on an ivsfm("IVLIML") fit vcov failed outright.
The scores are taken by central differences with a per-parameter step, there being no analytic gradient for most of these models. Where one does exist (NHN) the two agree to about , and they agree with numDeriv to the same order.
object$opt$value is
the OBJECTIVE the optimizer minimized, which is the negative summed
log-likelihood for most but not all estimators here, and
object$coefficients is what the fit REPORTS, which is not always what
it estimated. logLik() therefore reads neither blindly: "df" is
the number of parameters ESTIMATED (length(object$opt$par) where that
exists), because ivsfm("IVLIML") maximizes over 11 and
reports 6; and the returned value is the PLAIN log-likelihood, because
lcsfm with penalty_c > 0 maximizes a penalized
likelihood and carries the unpenalized one separately.
sfm's robust
divergence estimators (robust = "mlqe", "psi" or
"mdpd") minimize a divergence rather than a negative log-likelihood,
so logLik() returns NA with a warning and AIC()/
BIC() propagate it. Their objective is not on the likelihood scale and
must not be compared with one; print() names it accordingly.
Exception: psfm()'s
"GTRE_SEQ1" and "GTRE_SEQ2" models are estimated by a
moment-based procedure and "SSFE" by fixed-effects (LSDV)
regression, not by maximum likelihood; none of the three carries an
$opt component, so calling logLik() (or, transitively,
AIC()/BIC()) on those fits returns NA with a warning
rather than erroring.
coef.sfareg returns a named numeric vector.
vcov.sfareg returns a numeric matrix.
logLik.sfareg returns an object of class "logLik".
nobs.sfareg returns a single integer (or NA).
print.sfareg, summary.sfareg, TIC, influence_sfa; the entry points listed under ‘Description’.
Implementation of the cross-sectional stochastic frontier model across an array of distributional assumptions for both v and u (user specified). For panel models, see the psfm() call.
sfm(formula, model_name = c("NHN", "NHN_Z", "NE", "NE_Z", "NR", "THT", "NTN", "NG", "NNAK", "NU", "NGE", "NLN", "NW", "tHN", "TSL", "NB", "NGB2"), data, maxit.bobyqa = 10000, maxit.nlminb = 500, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 2, pgtol = 0, start_val = FALSE, PSopt = FALSE, use.nlminb = "auto", use.bobyqa = "auto", optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B", robust = c("mle", "mlqe", "psi", "mdpd"), c_mlqe = 0.2, eta = 0.01, alpha = 0.2, verbose = FALSE, start_from = NULL, Nsim = "auto", z_link = c("sd", "var"), vhet = NULL, uhet = NULL, muhet = NULL, scaling = NULL, shapehet = NULL, weights = NULL, wscale = TRUE, sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, sim_burn = NULL, sim_scrambling = 0L, sim_prime = NULL, sim_seed = NULL, rand.psoptim = NULL, keep_objective = FALSE, estimator = c("mle", "cols", "mols", "acols", "cmle"), cols_boot = 0, rand.cols = NULL)sfm(formula, model_name = c("NHN", "NHN_Z", "NE", "NE_Z", "NR", "THT", "NTN", "NG", "NNAK", "NU", "NGE", "NLN", "NW", "tHN", "TSL", "NB", "NGB2"), data, maxit.bobyqa = 10000, maxit.nlminb = 500, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 2, pgtol = 0, start_val = FALSE, PSopt = FALSE, use.nlminb = "auto", use.bobyqa = "auto", optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B", robust = c("mle", "mlqe", "psi", "mdpd"), c_mlqe = 0.2, eta = 0.01, alpha = 0.2, verbose = FALSE, start_from = NULL, Nsim = "auto", z_link = c("sd", "var"), vhet = NULL, uhet = NULL, muhet = NULL, scaling = NULL, shapehet = NULL, weights = NULL, wscale = TRUE, sim_type = c("halton", "sobol", "torus", "uniform"), antithetics = FALSE, sim_burn = NULL, sim_scrambling = 0L, sim_prime = NULL, sim_seed = NULL, rand.psoptim = NULL, keep_objective = FALSE, estimator = c("mle", "cols", "mols", "acols", "cmle"), cols_boot = 0, rand.cols = NULL)
formula |
a symbolic description for the model to be estimated |
model_name |
Which distributional pair to fit. The noise term
Matching ignores case, so |
data |
A data frame containing the variables named in |
maxit.bobyqa |
Maximum number of iterations for the bobyqa optimization routine |
maxit.psoptim |
Maximum number of iterations for the psoptim optimization routine |
maxit.optim |
Maximum number of iterations for the optim optimization routine |
REPORT, trace, pgtol, upper
|
Accepted but not used by |
start_val |
starting value (optional) |
PSopt |
use psoptim optimization routine (T or F) |
maxit.nlminb |
Maximum number of iterations for the nlminb optimization routine. |
use.nlminb |
|
use.bobyqa |
|
optHessian |
Logical. Should a numerically differentiated Hessian matrix be returned while using the optim routine? (for optim routine) |
inefdec |
Production or cost function |
Method |
Method passed to |
robust |
One of |
c_mlqe |
Numeric, default |
eta |
Numeric, default |
alpha |
Numeric, default |
verbose |
Logical. Print optimization progress messages? Default is |
weights |
Optional observation weights, one per row of the data actually fitted (that is, after missing-data handling; a length mismatch is an error rather than being recycled). They multiply the per-observation log-likelihood contributions. For frequency weights this is exact – a weight of 2 gives the same estimates as including the row twice, which the package tests – and for sampling weights it yields a pseudo-likelihood whose point estimates are consistent but whose Hessian-based standard errors are not. In that case use the sandwich methods this package registers, e.g. |
wscale |
Rescale |
shapehet |
Experimental. Optionally, a one-sided formula parameterizing the shape of |
scaling |
Optionally, a one-sided formula of covariates imposing the scaling property of Wang and Schmidt (2002) and Alvarez, Amsler, Orea and Schmidt (2006): |
start_from |
Optionally, a previously fitted |
Nsim |
Total number of simulation draws per observation for the simulated-maximum-likelihood models ( |
rand.psoptim |
Integer. seed for replication of psoptim. Default to |
keep_objective |
Logical. Store the likelihood function on the fitted object, so that |
estimator |
Two further values belong to the wrong-skewness suite (see sfa-wrongskew) and are implemented for
See ‘Details’. |
z_link |
Scale on which the variance-determinant linear predictor is placed, for the |
vhet |
One-sided formula for the determinants of the noise scale, e.g. |
uhet |
One-sided formula for the determinants of |
muhet |
One-sided formula for the determinants of the pre-truncation mean of |
sim_type |
Sequence used to build the simulated-ML draws, for the models fitted by simulation ( |
antithetics |
Logical. If |
sim_burn |
Number of leading elements of the sequence discarded. Default |
sim_scrambling |
Integer 0–3, |
sim_prime |
Prime for |
sim_seed |
Seed for the draws, for reproducibility. The caller's random-number stream is restored afterwards. |
cols_boot |
Integer. Number of nonparametric bootstrap resamples used to obtain standard errors under |
rand.cols |
Integer. Seed for the COLS bootstrap, for reproducibility. The caller's random-number stream is restored afterwards. Default |
sfm() fits a cross-sectional frontier (production; for a cost frontier) by maximum likelihood unless stated otherwise. Pick the pair of distributions for the noise and the inefficiency :
model_name |
noise / inefficiency |
use it when |
"NHN" |
normal / half-normal | the default; the standard frontier |
"NE" |
normal / exponential | a common alternative to "NHN" |
"NTN" |
normal / truncated normal | inefficiency with a non-zero mode; with muhet, Battese-Coelli (1995) |
"NHN_Z", "NE_Z" |
as above, |
covariates drive the scale of inefficiency |
"NG", "NNAK" |
normal / gamma, Nakagami | a flexible shape for |
"NR", "NU", "NGE", "TSL" |
normal / Rayleigh, uniform, generalized exponential, truncated skew-Laplace | alternative one-sided shapes |
"NLN", "NW" |
normal / lognormal, Weibull | simulated maximum likelihood |
"NGB2" |
normal / generalized beta II | one fit that nests most of the above |
"tHN", "THT" |
Student / half-normal, half |
heavy-tailed noise |
"NB" |
normal / binomial | wrong skewness read as widespread inefficiency; moments only |
Each is described in ‘Notes on individual models’ below. Other estimators (estimator = "cols", "acols", "cmle") and the robust divergences (robust) are described in their own subsections.
Modified ordinary least squares (estimator = "cols" or "mols"). The moment estimator of Olson, Schmidt and Waldman (1980, Journal of Econometrics), standardly called MOLS. Note the name: this is not Winsten's corrected OLS, which shifts the intercept by the largest residual and estimates no variance parameters at all. The package has shipped this estimator under the spelling "cols" since it was introduced, so that spelling continues to work and is not deprecated; "mols" is accepted as an exact synonym so the method can be found under its usual name. OLS is consistent for the slopes of a composed-error frontier whatever the one-sided distribution; only the intercept is biased, by , because the composed error does not have zero mean. COLS therefore keeps the OLS slopes, inverts the central moments of the OLS residuals for the scale parameters, and shifts the intercept up by the implied . Writing , for the second and third central moments of : for "NHN", and with ; for "NE", and with ; for "NG", from the third and fourth cumulants, and with .
No optimizer runs and the answer is deterministic, which makes COLS a natural robustness check against a maximum-likelihood fit that may have settled at a local optimum. It is less efficient than ML when the distributional assumption holds, and the "NG" inversion in particular leans on a fourth cumulant and is noticeably noisier than the other two at moderate sample sizes.
Wrong skew under COLS. A production frontier implies . When a sample comes out with the moment equations have no admissible solution: sfm() warns, reports sigu as 0 with the whole residual variance assigned to sigv, and returns no efficiency predictions. Read that as no evidence of inefficiency in these data, not as an estimate of zero. This is the Type I failure of Olson, Schmidt and Waldman and is common in small samples; $wrong_skew and $residual_moments record it.
The wrong-skewness suite (estimator = "acols" and "cmle"). Both address the same defect from opposite directions, and neither changes the model. Waldman (1982) showed that when the OLS residuals of a Normal-Half-Normal frontier are skewed the wrong way – positively, for a production frontier – the OLS fit is a local maximum of the likelihood at , so the estimate says there is no inefficiency anywhere in the industry. Nothing is wrong with the data when this happens: Simar and Wilson (2009) point out that wrong skew arises routinely when is small relative to , entirely within the assumed model. "acols" removes the third moment from the moment estimator, since that is the only ingredient which can produce the boundary; "cmle" keeps the likelihood but adds two constraints the assumed density implies, which the boundary does not satisfy. See sfa-wrongskew for the other options the package offers and how they differ.
The variance-determinant scale. sfm's _Z models place on the standard deviation, , while psfm's TRE_Z/GTRE_Z place it on the variance, , as do the competing packages. Since , fitting the same data under the two conventions returns the same , the same log-likelihood and the same marginal effects, but under the SD link is exactly half under the variance link. Reading a from one family as though it came from the other therefore doubles or halves every reported effect. z_link exists so the two can be aligned; the default preserves sfm's own convention so existing results do not move.
Heteroskedasticity in more than one component. vhet, uhet and muhet let , and (for "NTN") the pre-truncation mean each depend on their own covariates, in any combination. They are named formulas rather than further pipe segments deliberately: pipe position already carries different meanings in different model families – the second segment is in ttsfm but in psfm's GTRE_Z – and a fourth position would not be readable. The existing | z segment keeps its meaning, so nothing that worked before changes.
Two reasons to reach for vhet in particular. Ignored heteroskedasticity in does not merely cost efficiency: Caudill, Ford and Gropper (1995) show that it biases the estimated frontier, because the composed error's mean depends on the scales. And a that varies with observed characteristics is one of the few specifications that can absorb apparent wrong skewness without forcing onto the zero boundary. muhet with model_name = "NTN" is the Battese and Coelli (1995) inefficiency-effects model, in which covariates shift the location of inefficiency rather than its dispersion; that is usually the model wanted when the question is “what makes firms inefficient”, since then has a sign that reads directly.
These fits carry every positive quantity on the log scale, so the parameter vector is unconstrained and no variance can be driven onto a boundary by the optimizer; that is also why this path uses nlminb followed by an unbounded BFGS rather than the bounded four-stage scaffold the other models use. Coefficients are reported prefixed by block – Zv., Zu., Zmu. – since the same covariate may appear in more than one. marginal_effects reads the block of these fits with no extra arguments. estimator = "cols" and the robust divergences are moment-based and homoskedastic respectively, so neither combines with these arguments; both refuse rather than silently ignoring them.
Outputs include E[exp(-u)|e] given by exp_u_hat, following Battese and Coelli (1988, JoE), where appropriate. As of this version exp_u_hat is also returned for NE, NTN, NU, NGE, NLN and NW, together with the Jondrow et al. (1982) point predictor u_hat = E[u|e]; previously the exponential and truncated-normal models returned no efficiency prediction at all.
NR is not NHN. "NR" places a Rayleigh distribution on , not a half-normal, and the two are separate families rather than reparameterizations of one another: the Rayleigh density carries an extra factor of , and no rescaling removes it. The practical consequence is that the standardized skewness each contributes to the composed error is a different constant, against the half-normal's , so a sample generated under one cannot be fitted by the other. is reported on the second-raw-moment convention , matching "NHN" and making the two directly comparable; the Rayleigh scale is then , with and . Starting values come from inverting these moments, since the Rayleigh skewness is a constant and so identifies outright. Use data_gen_cs()'s y_pcs_r column to test it, not y_pcs.
tHN is not THT. In THT (Tancredi 2002) a single scale mixture is shared by both error components, so the noise and the inefficiency are both Student's t with the same degrees of freedom, and that shared mixture is what makes the composed error a closed-form skew-t. In tHN the noise is Student's t and the inefficiency is half-normal, drawn independently, so the two components have different tail behaviour, there is no closed form, and the density is the convolution evaluated by Gauss-Legendre quadrature. tHN is the heavy-tailed-noise model, which is what makes it the natural parametric comparison for the density-power robust estimators (robust = "mlqe", "psi", "mdpd"); THT cannot play that role because its inefficiency term is heavy-tailed too. Parameters are reported as (sigv, sigu, nu), the conventional order, not THT's inverted (sigu, sigv, a).
Two properties of tHN are worth knowing before using it. First, the degrees of freedom are weakly identified: on data simulated from the model at with a true , the profile log-likelihood varies by only about 0.24 across from 10 to 100 and peaks near 20, so a single reported should not be taken at face value. Profile over a grid of fixed and report the path. Because of that flat ridge sfm() always fits tHN from several widely separated starting values, keeps the best, and records the outcome in thn_starts; it warns when they reach different optima. Second, on real data can collapse onto zero, the heavy noise tail absorbing the entire one-sided component and leaving mean predicted efficiency near one. That is a property of the model rather than a numerical failure, so it is reported rather than suppressed: sfm() warns and sets thn_sigma_u_at_bound. Fits are also substantially slower than the closed-form models, since every likelihood evaluation is a quadrature whose node count grows with .
"NU" assumes (Li 1996; Nguyen 2010), giving the closed-form composed density . The reported theta is the upper bound of the inefficiency support, so predicted inefficiency is bounded by construction. "NGE" assumes a generalized exponential , whose composed density is a difference of two exponentially-tilted normal tail terms; it nests the exponential model and permits a non-monotonic inefficiency density. NGE reports sigu on the same scale as NE (the exponential mean, ) so the two are directly comparable.
"NLN" ( lognormal, reported as sigu = sdlog and mu = meanlog) and "NW" ( Weibull, reported as sigu = scale and k = shape) have no closed-form composed density and are estimated by simulated maximum likelihood. The integral is a product of two densities, and simulating from either one alone fails wherever the other is the narrow factor: draws from miss the normal kernel for a very inefficient unit, while draws from the kernel miss a sharply peaked . Half the draws are therefore taken from each, and combined by the balance heuristic of Veach and Guibas (1995), which weights every draw by the proposal likelier to have produced it. This needs no rule for choosing between the two and remains accurate in both regimes. The draws are generated once and reused at every optimizer iteration, so the objective is deterministic and reproducible; increase Nsim to reduce simulation error.
The efficiency predictor reuses the weights the likelihood computed, so u_hat is the Bayes rule under exactly the density that was maximised.
Simulated-ML draws. "NLN" and "NW" have no closed-form composed density and are fitted by maximum simulated likelihood, averaging over draws built by sim_type. Two properties are enforced whatever the sequence. Each observation receives its own contiguous block of the sequence rather than a strided subsequence: a column-major fill hands observation the elements of a van der Corput sequence, which is not equidistributed – at and Nsim = 100 the first observation's block spanned only , integrating over a quarter of the inefficiency distribution. And the draws are clamped away from 0 and 1, since qnorm() of an exact endpoint is infinite.
Blocking by observation is also what makes the draws worth using: Train (2002, ch. 9) attributes the advantage of a low-discrepancy sequence to its coverage and to the negative correlation it induces across observations, and the second only arises when observations receive different blocks. On draw counts, Bhat (2001) found 100 Halton draws more precise than 1000 pseudorandom ones, with 125 Halton draws giving half the simulation error of 1000 random draws; Nsim = "auto" scales the count with and warns if a supplied count is too small.
"NGB2": the generalized beta of the second kind. Makiela and
Mazur (2022), section 3, with density
reported as (sigv, sigu, nu, psi, tau). is a scale,
governs tail thickness (and so which moments exist, like the
degrees of freedom of a ), and and are shape.
The point of four parameters on one error component is what they nest. The limits are most of the rest of this table, so one fit says which of them the data want rather than a column of separate fits saying it by their likelihoods:
| restriction | distribution |
|
generalized gamma |
, , |
half-normal ("NHN") |
, |
exponential ("NE") |
, |
gamma ("NG") |
, |
Weibull ("NW"), at scale |
, |
half Student with degrees of freedom |
|
Lomax (Pareto II), shape and scale
|
The half- and Lomax cases are exact; the others are limits, and the
tests check that the error against each falls like rather than
merely that it is small at one value.
Note how the Lomax row sits against the exponential row above it: they carry
the same restriction and differ only in whether
is sent to infinity or left free, exactly as the half- row
is the finite- version of the half-normal row. So "NGB2"
already reaches the heavy-tailed one-sided family that a separate
Lomax model would provide, and no such model_name is needed.
One caveat specific to the Lomax region. A Lomax with shape
has no finite mean, and with no
finite variance. A "NGB2" fit that settles near
with a small is therefore in a region where does not
exist, which is what exp_u_hat and the JLMS predictor are estimating.
The fit itself is well defined – the likelihood only needs the density – but
inefficiency summaries from it should not be read as estimates of a population
mean that is not there.
has no interior maximum once the tail is not heavy, and a
large fitted value is not an estimate. Past a few hundred the density has
stopped moving – the discrepancy from the generalized-gamma limit is 2e-4 at
and falls only as – so the likelihood is flat
there. sfm() warns above 200; report that as “generalized gamma,
no heavy tail detected”, treat the standard error on as
meaningless, and read and , which still carry shape in
that limit. Starting values are placed at the half-normal point of the family
(, , ), so a fit that moves away
from it has moved for a reason.
The composed density has no closed form and is evaluated by deterministic
quadrature with two node sets, one at the quantiles of and one at the
quantiles of the noise truncated to , combined by the balance
heuristic. Either alone fails: the first misses a narrow noise, the second a
concentrated , and a four-parameter family can be either. Measured
against adaptive integration over 243 parameter settings, at points where the
density is not negligible () the median error in
is 1.2e-8 and the worst is 3.4e-2. The worst cases are one corner
of the grid – against a spiky generalized gamma
() – where adaptive integration disagrees with itself by up to
0.9, so it is not a reference that would justify more nodes. A single node set
on 's probability scale was wrong by up to 12 at the peak of the
density on the same grid.
It is not estimated by simulated maximum likelihood, and the reason is
worth knowing before reaching for simulation elsewhere. Written that way first,
with 100 draws per proposal, the simulated likelihood was biased upward and the
optimizer chased the bias: on generated from a half-normal it
walked to , – a near point mass, where
importance sampling from the inefficiency badly overstates the density – and
reported a log-likelihood of -336.9 where the exact value at those parameters
is -458.7. Simulated ML rewards whatever parameters maximise its own simulation
error, and four shape parameters give it room to find them.
This is nonetheless the slowest sfm() model: about 0.08 s per likelihood
evaluation at , so a fit takes minutes rather than seconds.
"NB": a binomial inefficiency, and what wrong skewness might
mean. Carree (2002). Every other one-sided distribution in this table is
positively skewed, so a production frontier implies a negatively skewed
composed error, and a positive residual skew has nowhere to go: the maximum
likelihood estimate is (Waldman 1982), read as
“no inefficiency”. Carree's observation is that this is not the only
reading. A binomial is negatively skewed for , and
that says something quite different – most firms carry considerable
inefficiency and only a few sit near the frontier. Same sample moment,
opposite economics.
The estimator inverts the second, third and fourth central moments of the OLS residuals (his Eq. 7),
whose ratio is a quadratic in
with roots .
Which root applies is settled by outside and by the sign
of inside it. n_bin and p_bin are reported alongside
sigu, which is so that it is comparable with the
other models.
Three things to know. First, a positive m3 is admissible here,
which is exactly what it is for; wrong_skew is set only when the
inversion genuinely fails. Second, it uses the fourth moment, so it is
markedly more fragile than the half-normal and exponential inversions – as
Greene's gamma estimator is, and for the same reason. Carree's infeasible
region is , where the implied is
negative; sfm() warns, reports no estimate, and that should be read as
evidence against a binomial inefficiency rather than as a number. It also
needs not to be small relative to , since
is then a small difference of large moments. Third, the
support is the integers: one unit of is one unit of the dependent
variable, so with a log dependent variable each “inefficiency category”
is a factor of . A scale factor is not identified from three moments
alongside , so there is none, and the implied efficiency
levels should be read with that in mind. The efficiency predictor is the exact
discrete posterior , with rounded to an
integer; n_bin keeps the unrounded moment estimate.
"TSL" assumes a truncated skew-Laplace inefficiency (Wang 2012, eq. 3.7; the distribution is due to Aryal and Rao 2005, and the model was published as Nguyen, Gupta and Wang 2014), with density for . This is a signed mixture of two exponentials – the second enters negatively – so it nests the exponential model as while allowing a non-monotonic inefficiency density, and the composed density is the corresponding signed combination of two normal-exponential terms. It is evaluated as a difference in log space rather than by subtracting the raw terms: both carry , which overflows once becomes small relative to , a region the optimizer routinely visits. lambda is a shape parameter and is the least sharply identified of the three. This is a property of the family, not of the fit, and it is worth understanding before reporting lambda at all. The shape parameter changes the standardised distribution remarkably little: equals 1 in both limits and , and its maximum over all is only , attained at ; the standardised skewness moves only from about 1.95 down to 1.79 and back. So the data identify the product far better than they identify and separately, and the two trade off against each other.
Measured on the package's own convergence design (, , 60 replications at each of ), estimating one parameter with the other held at its true value:
| estimated | log-MSE on log-n slope | sd at n = 4000 |
, free |
-0.65 | 0.038 |
, fixed |
-1.12 | 0.020 |
, free |
-0.38 (median) | 26.3 |
, fixed |
-1.07 (median) | 0.62 |
Each parameter attains the root- rate when the other is known and fails it when the other is free, which is the signature of weak scale–shape separation rather than of an optimiser problem. 's dispersion falls fortyfold once is fixed.
What to do about it. Report the functionals rather than the primitives where you can: converges at the root- rate even with both parameters free (slope -1.03, sd 0.031 at ), and nearly so (-0.89). Both are what an efficiency analysis actually uses. If itself is of interest, prefer profile-based inference over its reported -value, which is a Wald statistic on a parameter whose sampling distribution is neither symmetric nor tightly concentrated.
robust)Robust divergence-based estimation (robust argument). For
model_name = "NHN", sfm() can maximize a robust divergence
objective instead of the ordinary log-likelihood, downweighting the
influence of outlying observations:
"mlqe": minimum L estimation (Lq, Ferrari and Yang
2010), power parameter c_mlqe (). Objective
.
"psi": Psi-likelihood (Basu et al.-type divergence family),
power parameter eta. Objective
, where
is evaluated numerically at the
current parameter values on every optimizer iteration.
"mdpd": minimum density power divergence (Basu et al. 1998),
power parameter alpha. Objective
.
All three reduce to "mle" as their power parameter approaches 0.
Psi and MDPD produce identical parameter point estimates for the
same power parameter () – the two objectives are a
positive rescaling of one another, , which does not change the argmax; both are exposed for
compatibility with the corresponding paper's three-method terminology, not
because they differ numerically. MLqE is a genuinely distinct estimator.
Because these are M-estimators rather than ordinary MLE, the naive
Hessian-inverse standard error (optHessian = TRUE's usual output) is
not statistically valid for them – it assumes the information-matrix
equality, which these objectives do not satisfy in general. When
robust != "mle", sfm() instead reports a sandwich-form
standard error, , where is the Hessian of the
summed robust objective (as already computed for the point estimate) and
is built from the per-observation gradient of that
objective (obtained via numerical differentiation,
jacobian). This requires optHessian = TRUE;
std.errors/t.values are NA otherwise, same as for
ordinary MLE.
Formulas verified against an independently audited reference implementation
(cross-checked by direct numerical integration and finite differences),
including the sandwich SE's own verification (the c -> 0 limit against the
ordinary MLE Hessian SE, and a Monte Carlo check against the empirical
spread of repeated fits). Currently only implemented for
model_name = "NHN" – other models will error clearly if
robust != "mle" is requested.
Note on default iteration counts: sfm()'s optimization defaults (thousands of iterations) are much higher than psfm()'s (100/10/10). This is intentional: sfm()'s likelihoods are closed-form (no simulation), so each objective-function evaluation is cheap and many default iterations cost little; several of psfm()'s models require simulated maximum likelihood over Halton draws, where each evaluation is far more expensive. See psfm's Details for more.
An object of class "sfareg" containing the following components:
out |
A matrix with parameter estimates, standard errors, and t-values. |
opt |
A list containing the optimization results from the final optimization procedure. |
total_time |
The total computation time for model estimation. |
start_v |
The starting values used in the optimization. |
model_name |
The name of the stochastic frontier model estimated. |
formula |
The formula used in the model specification. |
exp_u_hat |
Predicted technical efficiency, E[exp(-u)|e]. Available for models: NHN, NHN_Z, NR, NG, NNAK, NE, NTN, NU, NGE, NLN, NW, NGB2, THT and tHN, and for |
u_hat |
Predicted inefficiency, E[u|e] (Jondrow et al. 1982). Available for NE, NTN, NU, NGE, NLN, NW, NGB2, THT and tHN. |
n_bin, p_bin
|
|
wrong_skew, sigma_u_at_bound, residual_m3
|
Wrong-skewness diagnostics. |
thn_sigma_u_at_bound |
Logical, |
ntn_sigma_v_at_bound |
Logical, |
thn_starts |
List, |
estimator |
|
wrong_skew |
Logical, |
boundary |
Logical, |
residual_moments |
The second, third and fourth central moments of the OLS residuals, |
cols_boot_draws |
The matrix of bootstrap replicates ( |
ng_starts |
Returned by |
sd_exp_u_hat |
Standard deviation of exp(-u) given the residual, following Tancredi (2002, section 2.2). Returned by |
med_u_hat |
Predicted technical efficiency (median values). Available only for the NHN model. |
coefficients |
A vector of estimated parameters. |
std.errors |
A vector of standard errors for the estimated parameters (NA if |
t.values |
A vector of t-values for the estimated parameters (NA if |
call |
The matched call. |
het |
Only present when |
sigma_u, sigma_v
|
Only present for a heteroskedastic fit: the fitted per-observation scales. |
v_spec, mu_spec
|
Only present for a heteroskedastic fit: the noise and pre-truncation-mean blocks, in the same shape as |
robust |
Only present for |
robust_c |
Only present for |
Standard errors require optHessian set to TRUE. When
robust != "mle" (model_name = "NHN" only), a sandwich-form SE
is reported instead of the ordinary Hessian-inverse one – see Details.
David H. Bernstein and Alexander Stead
Parmeter, C. F. and Zhao, S. (2023). An alternative corrected ordinary least squares estimator for the stochastic frontier model. Empirical Economics 64(6), 2831-2857. doi:10.1007/s00181-023-02401-1
Zhao, S. and Parmeter, C. F. (2022). The “wrong skewness” problem: Moment constrained maximum likelihood estimation of the stochastic frontier model. Economics Letters 221, 110901. doi:10.1016/j.econlet.2022.110901
Waldman, D. M. (1982). A stationary point for the stochastic frontier likelihood. Journal of Econometrics 18(2), 275-279. doi:10.1016/0304-4076(82)90041-0
Simar, L. and Wilson, P. W. (2009). Inferences from cross-sectional, stochastic frontier models. Econometric Reviews 29(1), 62-98. doi:10.1080/07474930903324523
sfa-wrongskew for the full wrong-skewness suite, psfm for panel models, zsfm for zero-inefficiency models, ttsfm for two-tier models, data_gen_cs for simulating data with known true parameters, and sfareg-methods for extracting results.
library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) ## Robust divergence-based estimation (NHN only): downweights the influence ## of outlying observations relative to ordinary MLE. std.errors/t.values ## are NA for these methods -- see Details. cs.nhn.mlqe <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NHN", data = cs_data_trial, robust = "mlqe", c_mlqe = 0.20) cs.nhn.psi <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NHN", data = cs_data_trial, robust = "psi", eta = 0.20) ## Heteroskedasticity in the noise as well as in the inefficiency: `| z` ## still drives sigma_u, `vhet` drives sigma_v. Coefficients come back ## prefixed by block (Zu., Zv.). cs.het <- sfm(formula = y_pcs_z ~ x1 + x2 | z, model_name = "NHN_Z", data = cs_data_trial, vhet = ~ z) ## Battese and Coelli (1995): covariates shift the LOCATION of inefficiency ## rather than its dispersion. Requires the truncated-normal family. cs.bc95 <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NTN", data = cs_data_trial, muhet = ~ z)library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) ## Robust divergence-based estimation (NHN only): downweights the influence ## of outlying observations relative to ordinary MLE. std.errors/t.values ## are NA for these methods -- see Details. cs.nhn.mlqe <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NHN", data = cs_data_trial, robust = "mlqe", c_mlqe = 0.20) cs.nhn.psi <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NHN", data = cs_data_trial, robust = "psi", eta = 0.20) ## Heteroskedasticity in the noise as well as in the inefficiency: `| z` ## still drives sigma_u, `vhet` drives sigma_v. Coefficients come back ## prefixed by block (Zu., Zv.). cs.het <- sfm(formula = y_pcs_z ~ x1 + x2 | z, model_name = "NHN_Z", data = cs_data_trial, vhet = ~ z) ## Battese and Coelli (1995): covariates shift the LOCATION of inefficiency ## rather than its dispersion. Requires the truncated-normal family. cs.bc95 <- sfm(formula = y_pcs ~ x1 + x2, model_name = "NTN", data = cs_data_trial, muhet = ~ z)
Fits several inefficiency distributions to the same data and combines their firm-level inefficiency predictions with data-driven weights. Where TIC and vuong answer "which distribution?", this answers what to do when the honest reply is "the data does not say" – selecting one specification and then reporting its efficiencies understates the uncertainty, because the standard errors condition on a choice that was itself estimated.
sfma(formula, data, models = c("NHN", "NE", "NTN", "NR", "NGE"), weights = c("sfma", "tic", "aic", "bic", "equal"), inefdec = TRUE, quiet = FALSE, ...)sfma(formula, data, models = c("NHN", "NE", "NTN", "NR", "NGE"), weights = c("sfma", "tic", "aic", "bic", "equal"), inefdec = TRUE, quiet = FALSE, ...)
formula, data, inefdec, ...
|
Passed to |
models |
Two or more |
weights |
How to weight the candidates. See ‘Details’. |
quiet |
Suppress the messages about dropped candidates, including a candidate that gets no |
The weighting schemes.
"sfma" is the criterion of Parmeter, Wan and Zhang (their eq. 13),
minimised over the unit simplex. The unobservable target is replaced by the prediction from the most general candidate, exactly as Mallows' replaces an unknown with the full model's estimate; without the penalty all weight would go to the largest candidate by construction. Their Theorem 1 says weights on models that nest the truth go to zero, and they do.
"tic", "aic" and "bic" are smoothed information-criterion weights in Buckland et al.'s (1997) form, . "tic" is the interesting one, because unlike AIC it does not assume the candidate is correctly specified – which is the premise of averaging in the first place. "equal" is the naive average, useful as a baseline.
"sfma" needs a unique most general candidate, and refuses without one. If every candidate has the same number of parameters the penalty is identical across them, so it drops out as a constant and the criterion collapses to "be as close as possible to " – minimised by putting weight 1 on whichever model was nominated as full. Measured on exponentially-distributed inefficiency with candidates "NHN", "NE" and "NR" (all with 5 parameters), that would have placed 0.879 on "NHN", the wrong distribution, where "tic" gave "NE" 0.934. Rather than break the tie silently, sfma errors and points at "tic". Including a genuinely more general candidate such as "NTN" makes the criterion well posed.
Reading the output. print reports the per-firm spread of efficiency across candidates. That spread is the reason to average: if the candidates agree, model choice was not costing anything and any single fit would have done; if they disagree, a single-model standard error is not describing the uncertainty a reader cares about.
Implementation. The weight problem is a convex quadratic program on the simplex, solved by projected gradient with an exact simplex projection (Duchi et al. 2008), so a weight that should be zero is exactly zero. It is checked against brute-force enumeration of the simplex in the tests rather than against itself.
An object of class "sfma" with components weights, u_hat (the averaged ), efficiency, u_by_model (the matrix of per-candidate predictions), fits, k, logLik and, for weights = "sfma", criterion.
Buckland, S.T., Burnham, K.P. and Augustin, N.H. (1997). Model selection: an integral part of inference. Biometrics, 53(2), 603–618.
Duchi, J., Shalev-Shwartz, S., Singer, Y. and Chandra, T. (2008). Efficient projections onto the l1-ball for learning in high dimensions. ICML.
Huang, C.J. and Lai, H.-p. (2012). Estimation of stochastic frontier models based on multimodel inference. Journal of Productivity Analysis, 38(3), 273–284.
Parmeter, C.F., Wan, A.T.K. and Zhang, X. (2019). Model averaging estimators for the stochastic frontier model. Journal of Productivity Analysis, 51(1), 91–103.
set.seed(3) n <- 300 x1 <- rnorm(n); x2 <- rnorm(n) ## True inefficiency is exponential. d <- data.frame(y = 1 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.4) - rexp(n, 1), x1 = x1, x2 = x2) ## Information-criterion weights: NE should earn most of the weight. a <- sfma(y ~ x1 + x2, data = d, models = c("NHN", "NE", "NR"), weights = "tic", quiet = TRUE) a a$weights ## The Parmeter-Wan-Zhang criterion needs a uniquely largest candidate. sfma(y ~ x1 + x2, data = d, models = c("NHN", "NE", "NTN"), weights = "sfma", quiet = TRUE)set.seed(3) n <- 300 x1 <- rnorm(n); x2 <- rnorm(n) ## True inefficiency is exponential. d <- data.frame(y = 1 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.4) - rexp(n, 1), x1 = x1, x2 = x2) ## Information-criterion weights: NE should earn most of the weight. a <- sfma(y ~ x1 + x2, data = d, models = c("NHN", "NE", "NR"), weights = "tic", quiet = TRUE) a a$weights ## The Parmeter-Wan-Zhang criterion needs a uniquely largest candidate. sfma(y ~ x1 + x2, data = d, models = c("NHN", "NE", "NTN"), weights = "sfma", quiet = TRUE)
Reports how much of a fitted parameter's uncertainty is simulation noise rather than sampling noise, by refitting the same data with independent randomizations of the draw sequence.
simulation_se(object, K = 10, seeds = NULL, quiet = FALSE, envir = parent.frame())simulation_se(object, K = 10, seeds = NULL, quiet = FALSE, envir = parent.frame())
object |
An |
K |
Number of refits, at least 2. |
seeds |
Optional integer vector of randomization seeds; its length
overrides |
quiet |
Suppress the per-refit progress messages. |
envir |
Environment in which to re-evaluate the stored call. The default is usually right; pass it explicitly if the data cannot otherwise be found. |
Simulated maximum likelihood replaces an integral with an average over draws.
A standard error from the Hessian answers “how much would this estimate
move in a new sample”, and says nothing about “how much would it move
with the same sample and different draws”. Randomizing the sequence
makes the second question answerable: refit times with independent
shifts and take the standard deviation across refits.
This is possible only because the draws are randomized by a uniform shift. An unrandomized Halton lattice returns the same answer every time, so the diagnostic would report exactly zero and mean nothing by it.
Reading the result. ratio is the share of the reported standard
error that comes from having simulated the integral. A large value is an
argument for raising Nsim, not for collecting more data. On the
package's reference design ("NW", , Nsim = 200) the
frontier slopes come back at about 0.5% while the variance and shape
parameters are near 8% – which independently reproduces the documented
behaviour that too few draws move everything except the slopes.
Quadrupling Nsim on that design cuts the simulation standard errors by
roughly a factor of six, faster than the factor of two that independent
pseudorandom draws would give. That is the low-discrepancy sequence earning its
place.
A data frame with one row per parameter: estimate, sampling_se
(from the Hessian), simulation_se (across refits) and their
ratio. Attributes n_refits and seed_arg record how many
refits succeeded and which argument was varied.
sfa_diagnostics for convergence diagnostics on a single fit.
## Kept deliberately small: every refit is a full simulated-ML fit, so the ## cost is K + 1 of them. K = 3 is the smallest that still shows the spread. d <- data_gen_cs(N = 200, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs_wb ~ x1 + x2, model_name = "NW", data = d, Nsim = 200) simulation_se(fit, K = 3, quiet = TRUE)## Kept deliberately small: every refit is a full simulated-ML fit, so the ## cost is K + 1 of them. K = 3 is the smallest that still shows the spread. d <- data_gen_cs(N = 200, rand = 1, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 1, mu = 0.5) fit <- sfm(y_pcs_wb ~ x1 + x2, model_name = "NW", data = d, Nsim = 200) simulation_se(fit, K = 3, quiet = TRUE)
Splits the third central moment of the composed error into the part contributed by the asymmetry of the inefficiency term, the part contributed by the asymmetry of the noise, and the part contributed by the dependence between them, following Bonanno, De Giovanni and Domma (2017).
skewness_decomp(object, nodes = 160L)skewness_decomp(object, nodes = 160L)
object |
A fitted model of class |
nodes |
Number of Gauss-Legendre nodes per dimension for the cross moments and the median. The default is where the FGM case stops moving in the ninth significant digit; raise it if a copula with strong tail dependence is in use. |
Write and . Then
and the last two terms vanish under independence. The function reports the three pieces as inefficiency (), noise () and dependence (the two cross terms together).
The point of the decomposition is what it says about the textbook model. A normal sets the noise term to zero, and independence sets the dependence term to zero, so under the standard specification the sign of the composed third moment is the sign of and nothing else. Every one-sided inefficiency distribution in common use is positively skewed, so a production frontier forces a negative composed skew, and a positively skewed residual has nowhere to go but . Skew in the noise and dependence between the components are perfectly ordinary explanations for the same residual, and the textbook model cannot express either. That is the sense in which Bonanno, De Giovanni and Domma call the wrong-skewness anomaly an ill-posed problem.
Also reported is Zenga's median-based measure, their Eq. (13),
whose sign is the one that says which way the density actually leans. When it disagrees in sign with the third central moment, print() says so: the mean-based measure is the misleading one in that case.
The marginal contributions are closed forms. The dependence term is computed by Gauss-Legendre quadrature on the unit square against the copula density, which covers every marginal and copula copsfm() supports rather than only the exponential/generalized-logistic/FGM combination for which the paper gives an explicit formula. The two agree to the quadrature's own accuracy. Note that the closed forms printed as Eq. (10) and Eq. (11) of the paper do not: their dependence terms disagree with a direct evaluation of the same expectations, and the corrected expressions – with the Gauss hypergeometric weights and – are the ones pinned in tests/testthat/test-wrongskew-suite.R.
An object of class "sfa_skew_decomp", a list with components:
components |
Named numeric of length three: |
total |
The third central moment of the composed error implied by the fit. |
share |
|
mean, median, variance
|
The first moment, median and variance of the composed error implied by the fit. |
median_based |
Zenga's measure, Eq. (13). |
model_name, udist, vdist, copula, copula_par
|
What the decomposition was taken of. |
Bonanno, G., De Giovanni, D. and Domma, F. (2017). The “wrong skewness” problem: a re-specification of stochastic frontiers. Journal of Productivity Analysis 47(1), 49-64. doi:10.1007/s11123-017-0492-8
Zenga, M. (1985). Statistica descrittiva. Turin: Giappichelli.
sfa-wrongskew for the other ways this package handles wrong skewness, copsfm for the model the decomposition was written for, skewness_test for a formal test on the residuals.
set.seed(21) n <- 300 d <- data.frame(x1 = rnorm(n), x2 = rnorm(n)) d$y <- 0.5 + 0.8 * d$x1 - 0.4 * d$x2 + rnorm(n, 0, 0.5) - abs(rnorm(n, 0, 1)) ## Under the textbook model two of the three components are identically zero. f <- sfm(y ~ x1 + x2, data = d, model_name = "NHN", estimator = "cols") skewness_decomp(f) ## With a skewed noise and a copula, all three are live. g <- copsfm(y ~ x1 + x2, data = d, copula = "fgm", udist = "exponential", vdist = "glogistic", n_nodes = 64) skewness_decomp(g)set.seed(21) n <- 300 d <- data.frame(x1 = rnorm(n), x2 = rnorm(n)) d$y <- 0.5 + 0.8 * d$x1 - 0.4 * d$x2 + rnorm(n, 0, 0.5) - abs(rnorm(n, 0, 1)) ## Under the textbook model two of the three components are identically zero. f <- sfm(y ~ x1 + x2, data = d, model_name = "NHN", estimator = "cols") skewness_decomp(f) ## With a skewed noise and a copula, all three are live. g <- copsfm(y ~ x1 + x2, data = d, copula = "fgm", udist = "exponential", vdist = "glogistic", n_nodes = 64) skewness_decomp(g)
Tests whether the OLS residuals of a stochastic frontier model are skewed in the direction the model implies. A production frontier has with , so the composed error is negatively skewed; residuals skewed the other way mean the maximum likelihood estimate of is exactly zero and every efficiency score is one.
skewness_test(object, test = c("agostino", "coelli"), alternative = c("auto", "less", "greater", "two.sided"))skewness_test(object, test = c("agostino", "coelli"), alternative = c("auto", "less", "greater", "two.sided"))
object |
An object of class |
test |
|
alternative |
|
What the tests are. Both start from the third central moment of the OLS residuals. Coelli's is asymptotically standard normal under the null of zero skewness. D'Agostino's transforms to approximate normality in finite samples rather than leaning on the asymptotic variance .
Why "agostino" is the default. It holds its size where the asymptotic form does not. Measured here, one-sided at a nominal 5% over 4,000 replications of symmetric residuals:
|
25 | 50 | 100 | 400 |
coelli |
0.031 | 0.037 | 0.043 | 0.050 |
agostino |
0.045 | 0.046 | 0.046 | 0.051 |
Coelli's is conservative below about ; the two agree by . The default also matches sfaR's. The implementation agrees with moments::agostino.test to within .
How to read the result. A small p-value under the default alternative is evidence for the frontier specification: the residuals are skewed the way inefficiency makes them. A large p-value is not evidence of a problem by itself – it may simply mean there is little inefficiency to detect, or too little data to detect it. What matters is the combination with wrong_skew: residuals skewed the wrong way and a fit reporting $sigma_u_at_bound is the Type I failure of Olson, Schmidt and Waldman (1980), in which case the boundary really is the maximum likelihood estimate (Waldman 1982) and the efficiency scores carry no information. See sfm.
An object of class "htest", so it prints like any other R test. Alongside the usual statistic, p.value and estimate (the sample skewness ) it carries m3, nobs, and wrong_skew – the last being TRUE when the third central moment is non-negative, matching sfm's own $wrong_skew.
Coelli, T. (1995). Estimators and hypothesis tests for a stochastic frontier function: a Monte Carlo analysis. Journal of Productivity Analysis, 6(3), 247–268.
D'Agostino, R.B. (1970). Transformation to normality of the null distribution of . Biometrika, 57(3), 679–681.
Olson, J.A., Schmidt, P. and Waldman, D.M. (1980). A Monte Carlo study of estimators of stochastic frontier production functions. Journal of Econometrics, 13(1), 67–82.
Schmidt, P. and Lin, T.-F. (1984). Simple tests of alternative specifications in stochastic frontier models. Journal of Econometrics, 24(3), 349–361.
Waldman, D.M. (1982). A stationary point for the stochastic frontier likelihood. Journal of Econometrics, 18(2), 275–279.
moment_range and spec_test for the Type II counterpart, sfm, sfa_diagnostics
cs <- data_gen_cs(N = 200, rand = 1, sig_u = 0.3, sig_v = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1) fit <- sfm(y_pcs_e ~ x1 + x2, model_name = "NE", data = cs) skewness_test(fit) # D'Agostino, one-sided skewness_test(fit, test = "coelli") # Coelli's M3T ## Also works on a bare residual vector. skewness_test(residuals(lm(y_pcs_e ~ x1 + x2, data = cs)))cs <- data_gen_cs(N = 200, rand = 1, sig_u = 0.3, sig_v = 0.4, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0.1) fit <- sfm(y_pcs_e ~ x1 + x2, model_name = "NE", data = cs) skewness_test(fit) # D'Agostino, one-sided skewness_test(fit, test = "coelli") # Coelli's M3T ## Also works on a bare residual vector. skewness_test(residuals(lm(y_pcs_e ~ x1 + x2, data = cs)))
Tests an assumed pair of distributions for the composed error using only the skewness and excess kurtosis of the OLS residuals. It needs no frontier fit, so it can be run before estimating anything and cannot be contaminated by an optimiser that failed. spec_test_all runs all twelve supported pairs and ranks them.
spec_test(object, noise = c("normal", "logistic", "laplace", "uniform"), inefficiency = c("halfnormal", "exponential", "genexponential"), null = c("bootstrap", "asymptotic"), B = 999, level = 0.05, seed = NULL) spec_test_all(object, level = 0.05)spec_test(object, noise = c("normal", "logistic", "laplace", "uniform"), inefficiency = c("halfnormal", "exponential", "genexponential"), null = c("bootstrap", "asymptotic"), B = 999, level = 0.05, seed = NULL) spec_test_all(object, level = 0.05)
object |
An |
noise |
Assumed distribution of |
inefficiency |
Assumed distribution of |
null |
How to get the p-value. |
B |
Bootstrap replications. Cheap here: nothing is fitted, so each replicate is |
level |
Size used for the reported verdict. |
seed |
Optional seed for the bootstrap. The RNG state is saved and restored. |
What it tests. Write for the share of composed-error variance contributed by . Papadopoulos and Parmeter (2021, eqs. 5 and 10) give
With both distributions assumed, , and are known constants, so the skewness alone identifies and the kurtosis equation has nothing left to fit. The statistic is the gap,
which is zero when the assumed pair is right. Its variance comes from the delta method applied to the central moments , which needs moments up to the eighth.
Why the default null is a bootstrap. The delta-method p-value is not reliable at usual sample sizes, and the failure is systematic rather than marginal. Rejection rates under correct specification, 300 replications per cell:
|
|
SNR | asymptotic | bootstrap |
| half-normal | 2000 | 1 | 0.087 | 0.050 |
| half-normal | 2000 | 2 | 0.103 | 0.057 |
| exponential | 2000 | 1 | 0.207 | 0.077 |
| exponential | 500 | 2 | 0.357 | 0.083 |
| exponential | 2000 | 2 | 0.307 | 0.060 |
Across all 24 cells the asymptotic null rejects 18.7% of the time at a nominal 5%; the bootstrap rejects 5.7%. The cause is the eighth moment: for a heavy-tailed it is estimated far too imprecisely at these sample sizes, and the variance comes out too small. The bootstrap studentises against the same statistic computed under the assumed pair, so it corrects the variance rather than merely relocating the statistic. Power is unaffected – against wrong pairs at it runs 0.93 to 1.00.
Type II failure. If the residuals are more skewed than the assumed can be – for instance under a half-normal. The pair is then refuted by the skewness alone, and spec_test says so instead of reporting a p-value that would suggest the question was close.
Reading spec_test_all. These are not sequential tests: no one of them depends on another's outcome, so the procedure implies no multiplicity correction. But picking the largest of twelve p-values is a selection, and the ranking should be read as comparative evidence rather than as a significance level.
Relation to skewness_test. That asks whether the residuals are skewed the way inefficiency makes them – a test of whether there is inefficiency. This asks whether the assumed distributions are the right ones. They answer different questions and are worth running together.
Provenance. The statistic here is reconstructed from PP2021 equations (5) and (10); the Supplementary Appendix in which PP2023 derive it was not available. The reconstruction is verified two ways in the tests: vanishes under correct specification across sixteen designs, and the tabulated constants and are reproduced exactly as and . PP2021's own equation (13) is deliberately not used, since PP2023 record that its terms are missing squares.
spec_test returns an object of class c("sfa_spec_test", "htest"), carrying statistic (the studentised ), p.value, Gamma, se, R, skewness, excess_kurtosis, type2_failure and reject. spec_test_all returns a data frame of all twelve pairs sorted by p-value.
Papadopoulos, A. and Parmeter, C.F. (2021). Type II failure and specification testing in the Stochastic Frontier Model. European Journal of Operational Research, 293(3), 990–1001.
Papadopoulos, A. and Parmeter, C.F. (2023). A specification test for the composed error term in the stochastic frontier model. Economics Letters, 233, 111390.
moment_range for the attainable ranges the same two equations imply, skewness_test, TIC, sfm
set.seed(2) n <- 2000 e <- rnorm(n, 0, 0.6) - abs(rnorm(n, 0, 1)) # normal / half-normal ## The true pair is not rejected. spec_test(e, noise = "normal", inefficiency = "halfnormal", B = 199, seed = 1) ## A wrong one is. spec_test(e, noise = "laplace", inefficiency = "exponential", B = 199, seed = 1) ## All twelve, ranked. spec_test_all(e)set.seed(2) n <- 2000 e <- rnorm(n, 0, 0.6) - abs(rnorm(n, 0, 1)) # normal / half-normal ## The true pair is not rejected. spec_test(e, noise = "normal", inefficiency = "halfnormal", B = 199, seed = 1) ## A wrong one is. spec_test(e, noise = "laplace", inefficiency = "exponential", B = 199, seed = 1) ## All twelve, ranked. spec_test_all(e)
Summary of a stochastic frontier fit, as returned by sfm, psfm, zsfm or ttsfm.
## S3 method for class 'sfareg' summary(object, ...)## S3 method for class 'sfareg' summary(object, ...)
object |
An object of class |
... |
Additional arguments passed to other methods |
Prints the call, the model name, and the estimated parameters with standard errors and t-values, together with the log-likelihood and the number of observations where the estimator provides them. Which parameters appear depends on the model: several report the , reparameterization rather than the raw scale parameters.
Invisibly returns its object argument unchanged, having printed the summary as a side effect.
David H. Bernstein
library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) summary(cs.nhnz)library(sfa) cs_data_trial <- data_gen_cs(N= 1000, rand = 1, sig_u = 0.3, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 4, mu = 1) cs.nhnz <- sfm(formula = y_pcs_z ~ x1 +x2| z, model_name = "NHN_Z", data = cs_data_trial, PSopt = TRUE) summary(cs.nhnz)
Model selection for stochastic frontier models when the candidates are not nested and none of them need be correctly specified. TIC is Takeuchi's information criterion, a version of AIC whose penalty is estimated from the data rather than assumed equal to the number of parameters. vuong is Vuong's pairwise test, which returns a p-value and is allowed to conclude that two specifications are indistinguishable.
TIC(object, detail = FALSE) vuong(object1, object2, correction = c("none", "aic", "tic"), level = 0.05)TIC(object, detail = FALSE) vuong(object1, object2, correction = c("none", "aic", "tic"), level = 0.05)
object, object1, object2
|
Objects of class |
detail |
If |
correction |
Bias correction applied to the log-likelihood difference before it is standardised. |
level |
Two-sided size used for the reported critical value and the verdict. |
Why not AIC. sfa offers many model_names for the same data, and AIC assumes the fitted model is correctly specified – exactly the assumption in doubt when choosing between "NHN", "NE", "NG" and "NTN". Most such pairs are not nested either, so the ordinary likelihood ratio test has no chi-square limit.
What TIC does. Takeuchi's criterion replaces AIC's penalty with , where is the sample Fisher information and the outer product of the per-observation scores. Under correct specification the information matrix equality gives , the trace collapses to , and TIC equals AIC. The gap between penalty and df – reported as ratio under detail = TRUE – is therefore a readable diagnostic in its own right: a ratio near one says the distributional assumption is not doing visible damage. Both penalty and df are counted on the scale the model is ESTIMATED on, which for ivsfm("IVLIML") is not the scale it reports: it maximises over 11 parameters and reports 6, and comparing the trace against the reported count made the ratio read near 2 when it is near 1.
What vuong does. It tests that the two models are equally close to the truth in the Kullback-Leibler sense. Neither model has to be correct. With the difference of per-observation log-likelihoods, the statistic is , asymptotically standard normal. Because it is a test rather than a criterion it can return "neither", which is frequently the honest answer and is not available from AIC.
Requirement. Both need per-observation log-likelihoods, so the fits must be made with keep_objective = TRUE. This is supported by sfm (for "NHN", "NE", "NR", "NG", "NNAK", "THT", "NTN", "NHN_Z", "NE_Z", "NU", "NGE", "NLN", "NW", "tHN" and "TSL"), by psfm for "GTRE" (estimator = "sml"), "TRE", "GTRE_Z", "TRE_Z", "GTRE_FML", "TFE_WMLE", "FD", "PL80", "PL80_MVTN", "BC92", "K1990" and "K1990modified" (from 1.2.1), whose contributions are per firm rather than per observation, and from 1.2.1 by zsfm, lcsfm, ttsfm ("TTNE", "TTHN"), copsfm, ivsfm ("IVLIML", "IVCF") and selsfm. vuong additionally requires both fits to use the same observations, which it checks by sample size and warns about when the data arguments differ.
Estimators that maximise nothing. psfm's "GTRE_SEQ1", "GTRE_SEQ2" and "SSFE", and ivsfm's "C2SLS", are moment-based or FE and have no log-likelihood, so no likelihood-based criterion applies to them and both functions refuse rather than returning a number built on NA.
TIC returns a single numeric value, or with detail = TRUE a list with components TIC, AIC, logLik, df, penalty and ratio.
vuong returns an object of class "sfa_vuong" with components statistic, p.value, n, lr, lr_adjusted, omega, correction, penalty, critical, level, models, favoured and exact_null. Lower TIC is better; favoured is the name of the winning model_name, or "neither".
Lai, H.-p. and Huang, C.J. (2010). Likelihood ratio tests for model selection of stochastic frontier models. Journal of Productivity Analysis, 34(1), 3–13.
Takeuchi, K. (1976). Distribution of informational statistics and a criterion of model fitting. Suri-Kagaku (Mathematical Sciences), 153, 12–18.
Vuong, Q.H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333.
sfm, skewness_test, sfa_diagnostics, AIC
set.seed(42) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) d <- data.frame(y = 1 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.4) - abs(rnorm(n)), x1 = x1, x2 = x2) a <- sfm(y ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) b <- sfm(y ~ x1 + x2, model_name = "NE", data = d, keep_objective = TRUE) ## Lower is better, and comparable across non-nested models. TIC(a) TIC(b) ## Is the penalty near the parameter count? If so, TIC has little to add. TIC(a, detail = TRUE)$ratio ## And a test, which may decline to pick a winner. vuong(a, b)set.seed(42) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) d <- data.frame(y = 1 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.4) - abs(rnorm(n)), x1 = x1, x2 = x2) a <- sfm(y ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) b <- sfm(y ~ x1 + x2, model_name = "NE", data = d, keep_objective = TRUE) ## Lower is better, and comparable across non-nested models. TIC(a) TIC(b) ## Is the penalty near the parameter count? If so, TIC has little to add. TIC(a, detail = TRUE)$ratio ## And a test, which may decline to pick a winner. vuong(a, b)
Refits a two-tier frontier from several starting values of , scores every solution on the same likelihood, and reports which is highest. Use it when ttsfm returns and you need to know whether that is the maximum likelihood estimate or a failure of the optimizer.
tt_boundary_report(object, data, sigma_v_starts = c(0.05, 0.3, 1), tol = 1e-3, tol_loglik = 1e-4, ...)tt_boundary_report(object, data, sigma_v_starts = c(0.05, 0.3, 1), tol = 1e-3, tol_loglik = 1e-4, ...)
object |
A |
data |
The data the model was fitted to. |
sigma_v_starts |
Positive starting values for |
tol |
Below this, the returned |
tol_loglik |
How much higher a restart's log-likelihood has to be before it counts as higher at all. Restarts landing on the same optimum differ in the last few digits; without this they would be reported as an improvement of zero log units. |
... |
Passed to |
This is a likelihood comparison and not a test. No p-value is reported and none should be inferred. The hypothesis a reader would naturally attach to it – – sits on the boundary of the parameter space, where the usual chi-square reference distribution for a likelihood-ratio statistic does not apply, and no appropriate null distribution has been established for this model. The output is a table of log-likelihoods and a statement about which is largest.
Why it is needed. "TTHN" drives to zero when and are close, and that collapse is the maximum likelihood estimate rather than a defect: with the difference is already symmetric and heavier-tailed than a normal, so a small normal underneath it is very nearly invisible. On a 2,000-observation draw at the profile in is flat to 0.000 log units from upward and decreasing above 0.01, with no interior maximum. Distinguishing that from an optimizer that merely stopped early is exactly what this function does.
The ordinary normal linear model on the same data is reported alongside, because a collapsed is not the same claim as “there is no frontier here”: on the design above the collapsed two-tier fit still beats the linear model by 9.6 log units, with both tiers estimated at 1.081 against truths of 1.0.
Cost. One ttsfm() refit per entry in sigma_v_starts. That is inexpensive for "TTNE" and expensive for "TTHN", where a single fit at 2,000 observations can take several minutes.
An object of class "sfa_tt_boundary", a list with components:
table |
A data frame with one row per start: the start, the |
best |
Which row attained the highest log-likelihood. Ties go to the returned fit. |
returned_is_best |
|
gap |
The best restart's log-likelihood less the returned fit's. |
collapsed |
|
lm_logLik, lm_gap
|
The ordinary linear model's log-likelihood, and how far the fit beats it. |
Polachek, S. and Yoon, B. J. (1987). A two-tiered earnings frontier estimation of employer and employee information in the labor market. The Review of Economics and Statistics 69(2), 296-302. doi:10.2307/1927239
ttsfm, and sfa_diagnostics for optimizer diagnostics generally.
d <- data_gen_cs(N = 1200, rand = 5, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0, sig_w = 0.5) f <- ttsfm(y_ttne ~ x1 + x2, data = d, model_name = "TTNE") tt_boundary_report(f, data = d, sigma_v_starts = c(0.05, 0.8))d <- data_gen_cs(N = 1200, rand = 5, sig_u = 1, sig_v = 0.3, cons = 0.5, beta1 = 0.5, beta2 = 0.5, a = 5, mu = 0, sig_w = 0.5) f <- ttsfm(y_ttne ~ x1 + x2, data = d, model_name = "TTNE") tt_boundary_report(f, data = d, sigma_v_starts = c(0.05, 0.8))
Fits the two-tier stochastic frontier model (2TSF): the composed error is a symmetric noise term plus two one-sided components pulling in opposite directions, (for example buyer and seller informational rents).
"TTNE": and exponential.
"TTHN": and half-normal.
"TTNLS": nonlinear least squares, with no distributional assumption. It estimates the frontier slopes only; see ‘Details’.
ttsfm(formula, model_name = c("TTNE", "TTHN", "TTNLS"), data, z_link = c("sd", "var"), maxit.bobyqa = 80000, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B", logit = TRUE, verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)ttsfm(formula, model_name = c("TTNE", "TTHN", "TTNLS"), data, z_link = c("sd", "var"), maxit.bobyqa = 80000, maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0, start_val = FALSE, PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B", logit = TRUE, verbose = FALSE, keep_objective = FALSE, rand.psoptim = NULL)
formula |
A symbolic description of the frontier, |
model_name |
|
data |
A data frame containing the variables named in |
z_link |
Scale on which the variance-determinant linear predictors are placed. |
maxit.bobyqa |
Maximum number of iterations for the bobyqa optimization routine. |
maxit.psoptim |
Maximum number of iterations for the psoptim optimization routine. |
maxit.optim |
Maximum number of iterations for the optim optimization routine. |
REPORT, trace, pgtol, upper
|
Accepted but not used by |
start_val |
|
PSopt |
Logical. Run the particle-swarm stage ( |
optHessian |
Logical. Finish with |
inefdec |
Has no effect in |
Method |
Method passed to |
logit |
Accepted but not used by |
verbose |
Logical. Print optimization progress messages? Default is |
keep_objective |
Logical. Retain the likelihood closure on the fitted object as |
rand.psoptim |
Integer seed for the particle-swarm stage, used only when |
The two-tier stochastic frontier model (Polachek and Yoon, 1987) writes the composed error as , with symmetric normal noise and , independent one-sided components: lowers below the frontier and raises it above. "TTNE" assumes and are exponential; "TTHN" assumes they are half-normal.
Determinants of and enter through the second and third parts of a two-pipe formula, y ~ x | zu | zw, which parameterize and through a log link (see z_link).
"TTNLS"Nonlinear least squares makes no distributional assumption, and the price is that it identifies only the frontier slopes and a composite intercept. , so the sum of squares cannot separate , and the intercept. ttsfm() therefore:
reports an intercept that absorbs , with standard errors for the intercept and slopes from the least-squares covariance . Without determinants TTNLS is ordinary least squares, and its coefficients and standard errors equal those of lm(); with determinants the standard errors are conditional on the fitted determinant coefficients;
returns NA for sigma_v, sigma_u and sigma_w, and for any determinant coefficients, with a warning;
returns metrics whose entries are all NA, since they are functions of those scales.
Use "TTNE" or "TTHN" when the individual scales are needed.
metrics
Like the other composed-error models in this package, "TTNE" and "TTHN" can have a genuine (non-error) boundary mode where a variance component is estimated near its numerical floor in some samples; spot-check fits before serious use. "TTHN"'s information-deficiency metrics (the metrics component) involve a large number of bivariate-normal-CDF evaluations and are more experimental than the parameter estimates themselves; they are NULL if that calculation fails.
Like sfm() and zsfm(), ttsfm()'s likelihoods are closed-form, so its optimization defaults are much higher than psfm()'s, which uses simulated maximum likelihood for several models. See psfm's Details.
An object of class "sfareg" containing the following components:
out |
A matrix with parameter estimates, standard errors, and t-values, one row per parameter. Scale parameters are reported on their natural scale and named, as of 1.2.1. The rows are Where A t-value on a scale parameter is not a conventional test. The null it corresponds to sits on the boundary of the parameter space, where the usual normal reference distribution does not apply. The column is reported for consistency with the rest of the package; read it as a rough magnitude, not as a p-value. |
opt |
A list containing the optimization results from the final optimization procedure. |
total_time |
The total computation time for model estimation. |
start_v |
The starting values used in the optimization. |
model_name |
|
formula |
The formula used in the model specification. |
coefficients |
A vector of estimated parameters. |
std.errors |
A vector of standard errors for the estimated parameters (NA if |
t.values |
A vector of t-values for the estimated parameters (NA if |
metrics |
A list of information-deficiency measures (M1, M2, M5, M6, M7, M10; see Polachek and Yoon) evaluated at the final parameter estimates. |
call |
The matched call. |
nobs |
The number of observations used in the fit. |
objective |
The likelihood closure, when |
Standard errors require optHessian = TRUE.
"TTHN" will drive to zero when and are close, and that is the maximum likelihood estimate rather than a failure of the optimizer. With the difference is already symmetric and heavy-tailed, so a small normal underneath it is very nearly invisible: on a 2,000-observation draw at the profile is flat to 0.000 log units from upward and decreasing above 0.01, with no interior maximum, and the collapsed fit beats the true parameter values by 1.29 log units. The two-tier structure is identified; the noise underneath two equal and opposite tiers is not. Asymmetric does not show this.
David H. Bernstein and Christopher F. Parmeter
Polachek, S. and Yoon, B.J. (1987). A Two-Tiered Earnings Frontier Estimation of Employer and Employee Information in the Labor Market. The Review of Economics and Statistics.
library(sfa) data <- data_gen_cs(N = 200, rand = 1, sig_u = 0.3, sig_v = 0.2, cons = 2, beta1 = 0.5, beta2 = 0.3, a = 5, mu = 0) eq <- y_pcs ~ x1 + x2 ttsfm(formula = eq, model_name = "TTNE", data = data)library(sfa) data <- data_gen_cs(N = 200, rand = 1, sig_u = 0.3, sig_v = 0.2, cons = 2, beta1 = 0.5, beta2 = 0.3, a = 5, mu = 0) eq <- y_pcs ~ x1 + x2 ttsfm(formula = eq, model_name = "TTNE", data = data)
The two-step test of Kim and Schmidt (2008): regress the JLMS predictor on firm characteristics and test the slopes, with the variance correction that makes the test valid.
uhet_test(object, z, data = NULL)uhet_test(object, z, data = NULL)
object |
An |
z |
A one-sided formula (e.g. |
data |
The data the model was fitted to. |
The procedure applied research actually runs is: fit the frontier, take
, regress it on , and test
the slopes. As usually performed that test is invalid. is
a generated dependent variable, so the first-step estimation error in
enters the second-step variance; the naive test omits it. Kim and
Schmidt show the omitted term is with
, so the naive test is valid only when
– which holds when is independent of , and
generally fails when they are correlated, the case applied work is usually in.
There is no distribution-free correction, which is why only the two cases the
paper works through are implemented.
What the correction buys. With independent of the two
tests agree, as the theory says they must. With strongly correlated
with () the naive standard error is 52% too
large at and the naive test's size collapses to 0.000 at
a nominal 5%, while the corrected standard error matches the true sampling
spread to within half a percent (ratio 0.995) and the test retains size 0.032.
A test that never rejects also has no power against local alternatives.
An honest caveat on size. The corrected test is conservative in
these simulations: on Kim and Schmidt's own base case (scaled exponential,
, , 1000 replications) it
rejects 0.024, 0.036 and 0.028 of the time at against
a nominal 0.05, where the paper reports 0.0495 for its corrected two-step test
at . The naive branch reproduces the paper's figure for the
uncorrected test closely (0.023–0.029 against their 0.0237). The
implementation was checked line by line against equations (5)–(13), the
first-step scores agree with an analytic half-normal score to
, is unbiased and Gaussian (kurtosis 2.92), and
an oracle test using the true sampling standard deviation has correct size
(0.045) – so the discrepancy is in the variance estimate, not in
or the reference distribution:
is 0.375, which compresses the ratio to a standard deviation of 0.93.
Two candidate explanations were tested and both refuted. Replacing the
robust second-step variance with the homoskedastic one – the difference
between the paper's “OLS” and “OLS-H” variants – changes
nothing (0.023/0.027/0.036 against 0.022/0.036/0.031). Nor does using the
Hessian information in place of the outer-product one when forming
(0.020/0.026/0.026). The discrepancy is therefore unexplained, not
merely unverified. Treat a rejection as meaningful and a non-rejection
as weak evidence.
Fits sitting on the wrong-skew boundary are refused rather than answered:
is then identical for every firm, so there is no estimated
inefficiency for to explain, and the information matrix is singular
there (Waldman 1982).
A data frame with the corrected and naive statistics, their common degrees of freedom and p-values. The second-step coefficients and both sets of standard errors are attached as attributes.
Kim, M. and Schmidt, P. (2008). Valid tests of whether technical inefficiency depends on firm characteristics. Journal of Econometrics 144, 409–427.
Wooldridge, J. M. (2002). Econometric Analysis of Cross Section and Panel Data, pp. 139–141.
sfm (its uhet argument fits the alternative
directly), marginal_effects, skewness_test
set.seed(2) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) z <- 0.9 * x1 + sqrt(1 - 0.81) * rnorm(n) # correlated with x: G != 0 y <- 0.5 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.5) - abs(rnorm(n, 0, 1)) d <- data.frame(y, x1, x2, z) f <- sfm(y ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) uhet_test(f, ~z, data = d)set.seed(2) n <- 400 x1 <- rnorm(n); x2 <- rnorm(n) z <- 0.9 * x1 + sqrt(1 - 0.81) * rnorm(n) # correlated with x: G != 0 y <- 0.5 + 0.5 * x1 + 0.5 * x2 + rnorm(n, 0, 0.5) - abs(rnorm(n, 0, 1)) d <- data.frame(y, x1, x2, z) f <- sfm(y ~ x1 + x2, model_name = "NHN", data = d, keep_objective = TRUE) uhet_test(f, ~z, data = d)
Panel data on U.S. investor-owned fossil fuel-fired steam electric utilities for the period 1986-1997. These data include measures of output, capital, labour and maintenance, and fuel.
data("USUtilities")data("USUtilities")
A data frame with 972 observations (a balanced panel of observations on 81 utilities over 12 years) on the following 7 variables.
firmIDa numeric vector containing a unique firm identifier
yeara numeric vector containing the year of the observation
qa numeric vector containing net steam electric power generation (MWh)
Ka numeric vector containing capital stock, calculated using a method described by Christensen and Jorgenson (1970)
La numeric vector containing quantity of labor and maintenance, calculated as cost divided by price index
Fa numeric vector containing quantity of fuel used, calculated as fuel costs divided by fuel price index
trenda numeric vector containing an annual time trend (1992=100)
The dataset covers 81 investor-owned utilities after aggregating subsidiaries and excluding plants in states with partial deregulation plans. Data sources include the Energy Information Administration (EIA), Federal Energy Regulatory Commission (FERC), and Bureau of Labor Statistics (BLS). Output is net steam electric generation from fossil fuel-fired boilers.
Rungsuriyawiboon, S. and Stefanou, S.E. (2007). 'Dynamic Efficiency Estimation: An Application to U.S. Electric Utilities.' Journal of Business & Economic Statistics, 25(2), pp. 226-238. doi:10.1198/073500106000000288
Christensen, L.R. and Jorgenson, D.W. (1970). 'U.S. Real Product and Real Factor Input, 1928-1967.' Review of Income and Wealth, 16(1), pp. 19-50. doi: 10.1111/j.1475-4991.1970.tb00695.x
data(USUtilities)data(USUtilities)
Fits the zero-inefficiency stochastic frontier model of Kumbhakar, Parmeter and Tsionas (2013): the sample is a mixture of fully efficient firms () and firms drawn from an ordinary normal-half-normal frontier.
"ZISF": the probability of being fully efficient is one constant for the whole sample.
"ZISF_Z": that probability depends on covariates, given in the second part of the formula (y ~ x | z).
zsfm(formula, model_name = c("ZISF", "ZISF_Z"), data, maxit.bobyqa = 10000,maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0,start_val = FALSE,PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B",logit = TRUE,verbose=FALSE, keep_objective = FALSE,rand.psoptim = NULL)zsfm(formula, model_name = c("ZISF", "ZISF_Z"), data, maxit.bobyqa = 10000,maxit.psoptim = 1000, maxit.optim = 1000, REPORT = 1, trace = 0, pgtol = 0,start_val = FALSE,PSopt = FALSE, optHessian = TRUE, inefdec = TRUE, upper = NA, Method = "L-BFGS-B",logit = TRUE,verbose=FALSE, keep_objective = FALSE,rand.psoptim = NULL)
formula |
A symbolic description of the frontier, |
model_name |
|
data |
A data frame containing the variables named in |
maxit.bobyqa |
Maximum number of iterations for the bobyqa optimization routine. |
maxit.psoptim |
Maximum number of iterations for the psoptim optimization routine. |
maxit.optim |
Maximum number of iterations for the optim optimization routine. |
REPORT, trace, pgtol, upper
|
Accepted but not used by |
start_val |
|
PSopt |
Logical. Run the particle-swarm stage ( |
optHessian |
Logical. Finish with |
inefdec |
Logical. |
Method |
Method passed to |
logit |
Logical, |
verbose |
Logical. Print optimization progress messages? Default is |
keep_objective |
Logical. Retain the likelihood closure on the fitted object as |
rand.psoptim |
Integer. Seed for replication of psoptim. Defaults to |
With probability a firm sits exactly on the frontier, so its composed error is noise alone, . With probability it is drawn from an ordinary normal-half-normal frontier. This is the natural specification when some fraction of the sample is thought to be fully efficient, which an ordinary frontier cannot represent: it forces almost surely.
Under "ZISF", is one constant, parameterized as . That parameterization is exactly symmetric in : and give identical fits and the optimizer may return either, so the sign of the reported gamma carries no meaning.
Under "ZISF_Z", varies with the covariates through the link chosen by logit:
logit = TRUE (the default): the logistic link , which covers .
logit = FALSE: , which is bounded above by 0.5. Under that link no observation can be assigned a probability of full efficiency above one half, whatever the data. It is not the probit analogue of the default and is retained only for backward compatibility.
Two per-observation predictions are returned, and they answer different questions:
post.prob is the posterior probability that the observation is fully efficient, given its residual.
jlms is the Jondrow et al. (1982) prediction conditional on the observation being in the inefficient regime. It is the ordinary normal-half-normal formula and does not weight by post.prob.
The prediction of over both regimes is (1 - post.prob) * jlms. The two can differ by orders of magnitude when the fitted is large, so use the product when you want itself.
Like sfm(), zsfm() has a closed-form likelihood, so its optimization defaults (thousands of iterations) are much higher than psfm()'s, which uses simulated maximum likelihood for several models. See psfm's Details.
An object of class "sfareg" containing the following components:
out |
A matrix with parameter estimates, standard errors, and t-values. Rows are |
opt |
A list containing the optimization results from the final optimization procedure. |
total_time |
The total computation time for model estimation. |
start_v |
The starting values used in the optimization. |
model_name |
|
formula |
The formula used in the model specification. |
jlms |
The Jondrow et al. (1982) inefficiency prediction |
post.prob |
Posterior probability that each observation is fully efficient. |
coefficients |
A vector of estimated parameters. |
std.errors |
A vector of standard errors for the estimated parameters (NA if |
t.values |
A vector of t-values for the estimated parameters (NA if |
call |
The matched call. |
nobs |
The number of observations used in the fit. |
objective |
The likelihood closure, when |
Standard errors require optHessian = TRUE.
Christopher F. Parmeter and David H. Bernstein
Kumbhakar, S.C., Parmeter, C.F. and Tsionas, E.G. (2013) 'A zero inefficiency stochastic frontier model', Journal of Econometrics, 172(1), pp. 66-76. doi:10.1016/j.jeconom.2012.08.021.
Jondrow, J., Lovell, C.A.K., Materov, I.S. and Schmidt, P. (1982) 'On the estimation of technical inefficiency in the stochastic frontier production function model', Journal of Econometrics, 19(2-3), pp. 233-238.
sfm, psfm, ttsfm, sfareg-methods, panel89
library(sfa) eqz <- y ~ q1 + q2 + q3 + q4 + q5 + w1 + w2 + w3 + w4 | z data(panel89) f <- zsfm(formula = eqz, model_name = "ZISF_Z", data = panel89, logit = TRUE) f ## Inefficiency over both regimes, not just the inefficient one head((1 - f$post.prob) * f$jlms)library(sfa) eqz <- y ~ q1 + q2 + q3 + q4 + q5 + w1 + w2 + w3 + w4 | z data(panel89) f <- zsfm(formula = eqz, model_name = "ZISF_Z", data = panel89, logit = TRUE) f ## Inefficiency over both regimes, not just the inefficient one head((1 - f$post.prob) * f$jlms)