Teddy Paper
📝 Article Type: Research Paper
Cross-Architecture Statistical Comparison of Machine Translation Output: SMT versus Six Large Language Models — Ziyan Wang, Yufan Duan, Youxi He (New Talent Academy, Beijing)
Keywords: SMT, LLM, Machine Translation, Mixed-effects Models,
Lexical Diversity, Stylometry
Introduction
Machine translation based on ML has become an indispensable part in
today’s world. In the past, translation systems were based on SMT, which
relies on no neural network infrastructure, generating translations
purely through approaches like parallel corpora, n-gram language models,
and beam-search decoding. Statistical machine translation and large
language models represent two fundamentally different approaches to the
translation task. SMT decomposes translation into discrete sub-problems
(word alignment, phrase extraction, and language model scoring), each
optimized independently (Koehn et al. 2007). Recently, however,
Transformer-driven systems become the mainstream, for they can capture
nuances and produce high-quality translation (Luong, Pham, and Manning
2015; Vaswani et al. 2017). LLMs learn a unified mapping from source to
target text through next-token prediction over billions of parameters
(Vaswani et al. 2017).
The question lies in this: how can we distinguish the subtle differences
between the texts from the two paradigms statistically? Despite this
divergence, existing comparisons have focused almost exclusively on BLEU
scores and human-rated adequacy, treating the output as a black-box
quality signal. Few studies examine whether these paradigms produce
different texts: differences in vocabulary choice, sentence structure,
or stylistic register that persist even when translations are judged
equally accurate. Such text-internal properties matter for downstream
applications: downstream models (e.g., information retrieval, authorship
attribution) respond to surface-level distributional shifts, and
translation consumers may respond differently to lexically sparse versus
diverse output regardless of semantic fidelity.
Past literatures that involved with stylometric studies focused on the
difference between LLM-generated text and human-written text
(Muñoz-Ortiz, Gómez-Rodrı́guez, and Vilares 2024; Reinhart et al. 2025).
Based on their methodology, we have conducted the first statistical
investigation focusing on the difference between texts generated by two
ML paradigms, where translated texts generated by traditional
phrase-based statistical machine translation (SMT) and six LLMs are the
subjects, including 833 translations (360 from SMT and 473 from LLMs).
The analysis has four layers: lexical diversity, sentence complexity,
sentiment polarity, and stylometric features. To eliminate possible
confounds, we use a factorial design (architecture
direction genre), employing mixed-effects models,
Kolmogorov–Smirnov tests, and SVM classification.
The SVM achieves accuracy, with lexical diversity (STTR Cohen’s
, SMT more diverse) as the largest divergence and sentence
length second (, LLM outputs 2.6 longer on average).
Genre proves a strong mediating factor when source text identity is
controlled. Only lexical diversity remains robust (,
); sentence-length and sentiment differences stem from
translation direction. These results provide empirical evidence for
stylistic divergence between translation architectures.
The remainder of this paper is organized as follows.
Section 2 describes the experimental design,
translation systems, feature extraction, and statistical procedures.
Section 3 presents findings.
Section 4 interprets results and outlines
limitations. Section 5 concludes.
Methods
Experimental Design
We used a factorial design with Architecture (SMT
vs. LLM), Direction (ZHEN vs. ENZH), and Genre (News
vs. Literature) as factors. A prospective power analysis for a two-sided
two-sample -test establishes the required sample size for detecting a
large effect ():
n \geq \frac{2(z_{1-\alpha/2} + z_{1-\beta})^2}{d^2}, \label{eq:power}$$ where $\alpha = 0.05$, $\beta = 0.20$ (power $= 0.80$). With $z_{0.975} = 1.96$ and $z_{0.80} = 0.842$, Equation [eq:power] yields $n \geq 25$ per group. The actual corpus | Architecture | Models/Variants | ZH$\to$EN | EN$\to$ZH | Total | | |:-------------|----------------:|----------:|----------:|------:|----:| | SMT | 5 | 200 | 160 | 360 | | | LLM | 6 | 233 | 240 | 473 | | | Total | — | 433 | 400 | 833 | | Factorial design with sample sizes ### Source Texts A total of 80 source texts (20 per category) were selected: Chinese news articles sourced from public Xinhua publications, Chinese literary excerpts drawn from five or more authors including Lu Xun, Wang Xiaobo, and Yu Hua; English news articles from publicly available sources; and English literary passages comprising original short works and excerpts from Project Gutenberg. Text length ranged from 200 to 800 words. All texts were stored in UTF-8 encoding. ## Translation Systems ### SMT The SMT system was a custom Python phrase-based pipeline following the noisy-channel decomposition: $$P(\mathbf{e} \mid \mathbf{f}) \propto P(\mathbf{f} \mid \mathbf{e}) \cdot P(\mathbf{e}), \label{eq:noisy}$$ where $\mathbf{f}$ is the source sentence, $\mathbf{e}$ the target sentence, $P(\mathbf{f} \mid \mathbf{e})$ the translation model, and $P(\mathbf{e})$ the language model. Word alignment was performed using IBM Model 2 and HMM alignment via `fast_align` (Dyer, Chahuneau, and Smith 2013), followed by grow-diag-final-and (gdfa) symmetrization. Phrase pairs were extracted with four features (direct and inverse phrase translation probabilities, direct and inverse lexical weighting). Decoding employed a beam-search with future cost estimation and recombination, guided by a Kneser-Ney smoothed 3-gram language model (Kneser and Ney 1995). The modified Kneser-Ney interpolation is: $$\begin{gathered} P_{\text{KN}}(w_i \mid w_{i-2}^{i-1}) = \frac{\max(c(w_{i-2}^{i}) - D, 0)}{\sum_{w'} c(w_{i-2}^{i-1} w')} \\ + \gamma(w_{i-2}^{i-1}) \cdot P_{\text{KN}}(w_i \mid w_{i-1}), \label{eq:kn} \end{gathered}$$ where $D$ is a learned discount parameter and $\gamma$ ensures normalization. The system was trained on the WMT news-commentary v12 corpus ( 50,000 sentence pairs). The primary alignment variant (fast_align $+$ gdfa) yielded 65,909 phrase pairs. ### LLMs Six decoder-only LLMs were accessed via commercial APIs. DeepSeek V4 Pro and DeepSeek V4 Flash were served through the DeepSeek API; GLM-5.1, GLM-5, Kimi K2.6, and Qwen3.6-Plus through OpenCode Go. Temperature was set to $T = 0.0$, seed to 42, and reasoning was disabled where available. A uniform prompt template was used (Appendix 7). Qwen3.6-plus completed 73 of 80 source texts; the remaining 7 were Chinese-to-English literary translations lost to API timeouts. Table 2 lists specifications. | Model | Provider | Context | Reasoning | $n$ | |:------------------|:-------------|:-------:|----------:|----:| | deepseek-v4-pro | DeepSeek API | 1M | Disabled | 80 | | deepseek-v4-flash | DeepSeek API | 1M | Disabled | 80 | | glm-5.1 | OpenCode Go | 203K | Disabled | 80 | | glm-5 | OpenCode Go | 128K | Disabled | 80 | | kimi-k2.6 | OpenCode Go | 256K | N/A | 80 | | qwen3.6-plus | OpenCode Go | 260K | N/A | 73 | LLM model specifications ## Feature Extraction Nineteen features were extracted across four dimensions (Table 3). **Lexical diversity** was measured using STTR (standardized type–token ratio), MTLD, and HD-D, computed with Jieba for Chinese tokenization. STTR is defined as the mean type–token ratio across overlapping 100-word windows with 50-word stride: $$\text{STTR} = \frac{1}{K} \sum_{k=1}^{K} \frac{|V_k|}{100}, \label{eq:sttr}$$ where $K$ is the number of windows and $|V_k|$ is the number of unique types in window $k$. Texts shorter than 100 words fall back to global TTR. **Sentence complexity** was captured by the mean, standard deviation, skewness, and kurtosis of sentence length, with sentence boundaries detected by regex. **Sentiment** was assessed using VADER (Hutto and Gilbert 2014) for English output and a custom keyword-based polarity lexicon (15 positive and 15 negative words per language) for Chinese output. The polarity score for each text is: $$\text{Pol} = \frac{N_{\text{pos}} - N_{\text{neg}}}{N_{\text{total}}} \in [-1, +1]. \label{eq:polarity}$$ **Stylometry** included character-type distribution entropy and function-word ratios (approximately 150 words for English, 100 for Chinese). Character-type distribution entropy measures diversity across four character categories (alphabetic $\alpha$, punctuation $\sigma$, digit $\delta$, other $\omega$): $$H(\mathbf{p}) = -\sum_{i \in \{\alpha,\sigma,\delta,\omega\}} p_i \ln p_i, \label{eq:entropy}$$ where $\mathbf{p} = (p_\alpha, p_\sigma, p_\delta, p_\omega)$ is the probability vector of character types. =10em | Dimension | Metrics | Tool | |:--------------------|:--------------------------------------------------------------|:---------------------------------------------------------------| | Lexical Diversity | STTR, MTLD, HD-D | Python, Jieba (Chinese tokenization) | | Sentence Complexity | Mean, SD, skewness, kurtosis of sent. len. | Regex-based sentence segmentation | | Sentiment | Polarity | VADER (Hutto and Gilbert 2014) (EN), keyword lexicon (ZH) | | Stylometry | Char.-type entropy, func. word ratio, alpha/punct/digit ratio | Custom word lists ($\sim$150 EN, $\sim$100 ZH) | Feature extraction summary ## Statistical Methods #### Mixed-effects model. The primary analysis used linear mixed-effects models with restricted maximum likelihood estimation via `statsmodels`. For each dependent variable, the model includes a random intercept for source text identity to account for the paired design: $$Y_{ij} = \beta_0 + \beta_1 S_{ij} + \beta_2 D_{ij} + \beta_3 G_{ij} + \beta_4 (S{:}D)_{ij} + \beta_5 (S{:}G)_{ij} + \beta_6 (D{:}G)_{ij} + \beta_7 (S{:}D{:}G)_{ij} + u_j + \varepsilon_{ij}, \label{eq:mixed}$$ where $S_{ij}$ is the SMT indicator, $D_{ij}$ the direction, $G_{ij}$ the genre, $S{:}D$ etc. denote interactions, $u_j \sim \mathcal{N}(0, \sigma_u^2)$ is the source-text random intercept, and $\varepsilon_{ij} \sim \mathcal{N}(0, \sigma^2)$ the residual. The primary test is $\beta_1$ (architecture main effect). #### Kolmogorov–Smirnov test. Distributional differences in sentence length between architectures are assessed at the text-pair level using the two-sample KS statistic: $$D = \sup_x |F_{\text{SMT}}(x) - F_{\text{LLM}}(x)|. \label{eq:ks}
Effect size.
All pairwise comparisons are reported with Cohen’s , using the pooled
standard deviation:
d = \frac{\mu_{\text{SMT}} - \mu_{\text{LLM}}}{s_p}, \qquad s_p = \sqrt{\frac{(n_1-1)s_1^2 + (n_2-1)s_2^2}{n_1 + n_2 - 2}}. \label{eq:cohensd}
SVM classification.
As multivariate convergent validation, a linear SVM with and
balanced class weights classifies architecture from the full feature
vector: $$\hat{y} = \operatorname{sign}(\mathbf{w}^\top \mathbf{x} + b),
\label{eq:svm}$$ with GroupKFold (, groups source text) to
prevent data leakage. Features are standardized to -scores within
each fold.
Multiplicity correction.
All -values across hypothesis tests are adjusted using the
Holm–Bonferroni procedure (Holm 1979), which controls the family-wise
error rate at : $$p_{(i)} \leq \frac{\alpha}{k - i + 1},
\label{eq:holm}$$ where are the
sorted -values and tests per family.
Results
Descriptive Statistics
Table 4 reports descriptive statistics.
Lexical diversity (STTR) showed the largest architecture effect: SMT
translations (mean = 0.952) exceeded LLM translations (mean = 0.745),
Cohen’s , Kolmogorov–Smirnov : near-complete
distributional separation. Mean sentence length produced a moderate
effect (, SMT 3.53 vs. LLM 9.33 words per sentence).
Sentiment polarity shifted positive in LLM translations ().
POS tag entropy and function word ratio showed negligible effects
(, ).
By direction, the STTR effect remained large for ZH EN
() but was inflated in EN ZH primarily due to a
segmentation artifact: Jieba tokenization of SMT Chinese output produced
single-token “sentences” (mean length 1.0), driving SMT STTR
to near ceiling (0.983). Sentiment polarity and function word ratio
registered 0.000 for both architectures in the EN ZH condition:
the English-trained sentiment model and English function word list
produced null outputs on Chinese text. These metrics were excluded from
EN ZH analyses in subsequent tests. The ZH EN-only STTR
effect () showed the architecture difference persisted
without this measurement confound.
| ZHEN | ENZH | |||||
|---|---|---|---|---|---|---|
| 2-3 (lr)4-5 (lr)6-7 Metric | SMT | LLM | SMT | LLM | KS | |
| STTR | 0.928 | 0.763 | 0.983 | 0.727 | 0.918 | |
| Mean sentence length | 5.56 | 17.90 | 1.00 | 1.01 | 0.490 | |
| Sentiment polarity | 0.127 | 0.597 | 0.000 | 0.000 | 0.387 | |
| POS entropy | 0.589 | 0.582 | 0.304 | 0.354 | 0.233 | |
| Function word ratio | 0.178 | 0.291 | 0.000 | 0.000 | 0.333 | |
| Alphabetic ratio | 0.784 | 0.807 | 0.903 | 0.897 | 0.211 |
Descriptive statistics and effect sizes: SMT vs. LLM
: ENZH sentiment and function word ratio are zero for
both architectures, a measurement artifact from sentiment model and
function word list operating on Chinese output. These metrics are
excluded from ENZH comparisons.
Table 5 breaks down effect sizes by
genre. The STTR architecture effect was larger in news ()
than in literature (), driven by near-ceiling SMT performance
on news texts (SMT mean STTR = 0.997). Sentiment polarity likewise
showed a genre asymmetry: news versus literature
, a ratio of 7.7:1. Mean sentence length effects were more
consistent across genres (),
length differences between architectures operate independently of text
domain. POS entropy showed a small genre reversal: negative in news
(), slightly positive in literature (); the
architecture effect on syntactic diversity, while small, may interact
with genre characteristics.
| Metric | News | Literature | |
|:---------------------|:--------:|:--------------:|:--------------------------------------:|
| STTR | | | 5.84 |
| Mean sentence length | | | 1.19 |
| Sentiment polarity | | | 7.71 |
| POS entropy | | | 1.83 |
| Function word ratio | | | 5.83 |
Genre-stratified effect sizes
Hypothesis Testing
Table 6 summarizes the four hypotheses
with Holm-corrected tests. H1, that LLM translations have lower lexical
diversity than SMT, was supported: Cohen’s , KS ,
and the mixed-effects model showed robustness
(, ). H2 predicted longer
sentences in LLM output; the univariate effect was moderate
(, KS ) but the mixed-model coefficient was
negligible (, ); the apparent length
difference is driven by source-text identity, not architecture. H3, that
LLM translations express more positive sentiment, showed a moderate
univariate effect () that vanishes in the EN ZH
direction (). H4 predicted lower POS tag entropy in LLM
output; the effect was small (, KS ) and reached
significance only through sample size. Only H1 was both statistically
and practically large.
| Metric | Cohen’s | KS | Robust? | |
|---|---|---|---|---|
| H1 | STTR (Lexical Diversity) | 0.918 | ||
| H2 | Mean Sentence Length | 0.490 | ||
| H3 | Sentiment Polarity | 0.387 | ||
| H4 | POS Tag Entropy | 0.233 |
Hypothesis test results (Holm-corrected)
: Large univariate effect, but mixed-model coefficient
(). H2 is not robust despite
significant KS test.
: Effect vanishes () in ENZH direction.
Mixed-effects Model Estimates
Table 7 reports mixed-effects model
coefficients for the architecture fixed effect, controlling source text
identity as a random intercept. Only STTR produced a substantial
coefficient: (). Mean sentence
length, sentiment polarity, and function word ratio each yielded
coefficients near zero (, ): their
apparent univariate effects are absorbed by source-text-level variation.
POS entropy reached statistical significance (,
) but the raw effect size () is negligible. STTR
is the sole metric that consistently differentiates SMT from LLM
translations when source text identity is modelled.
| Metric | (SMT) | Significant? | |
|---|---|---|---|
| STTR | Yes | ||
| Mean sentence len. | No | ||
| Sentiment polarity | No | ||
| POS entropy | Yes | ||
| Function word ratio | No |
Mixed-effects model: architecture main effect (pooled, )
: Statistically significant but effect size negligible (raw
).
SVM Classification
We trained a linear SVM with GroupKFold cross-validation (,
grouped by source text) on standardized features.
Figure 1 shows the confusion matrix;
Figure 2 plots feature coefficients. The
classifier achieved 97.96% accuracy (), F1 98.25%
(), and MCC 0.959 (). The normalized feature
weight vector from the linear decision boundary reveals the
discriminative contribution of each dimension:
\mathbf{w} = (-2.18_{\text{STTR}}, +1.89_{\text{Len}}, -0.07_{\text{Pol}}, -0.80_{\text{Ent}}, -1.22_{\text{FW}}).
\label{eq:svm-weights}$$ STTR carried the largest negative weight
($-2.18$, SMT), followed by mean sentence length ($+1.89$, LLM).
Function word ratio ($-1.22$) and POS entropy ($-0.80$) contributed
moderately; sentiment polarity was negligible ($-0.07$).
Mean sentence length’s prominence in the SVM (weight $+1.89$) contrasts
with its null mixed-model coefficient ($\beta{=}-0.0009$, $p{=}0.996$).
The SVM exploited sentence length patterns tied to specific source
texts, a signal absorbed by the random intercept in the mixed model. The
classifier’s performance thus partly reflects source-level regularities,
not pure architecture properties.
