I18N_Arabic

# Source for file AutoSummarize.php

Documentation is available at AutoSummarize.php

1. <?php
2. /**
3. * ----------------------------------------------------------------------
4. *
5. * Copyright (c) 2006-2016 Khaled Al-Sham'aa.
6. *
7. * http://www.ar-php.org
8. *
9. * PHP Version 5
10. *
11. * ----------------------------------------------------------------------
12. *
13. * LICENSE
14. *
15. * This program is open source product; you can redistribute it and/or
16. * modify it under the terms of the GNU Lesser General Public License (LGPL)
17. * as published by the Free Software Foundation; either version 3
18. * of the License, or (at your option) any later version.
19. *
20. * This program is distributed in the hope that it will be useful,
21. * but WITHOUT ANY WARRANTY; without even the implied warranty of
22. * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
23. * GNU Lesser General Public License for more details.
24. *
25. * You should have received a copy of the GNU Lesser General Public License
26. * along with this program. If not, see <http://www.gnu.org/licenses/lgpl.txt>.
27. *
28. * ----------------------------------------------------------------------
29. *
30. * Class Name: Arabic Auto Summarize Class
31. *
32. * Filename: AutoSummarize.php
33. *
34. * Original Author(s): Khaled Al-Sham'aa <khaled@ar-php.org>
35. *
36. * Purpose: Automatic keyphrase extraction to provide a quick mini-summary
37. * for a long Arabic document.
38. *
39. * ----------------------------------------------------------------------
40. *
41. * Arabic Auto Summarize
42. *
43. * This class identifies the key points in an Arabic document for you to share with
44. * others or quickly scan. The class determines key points by analyzing an Arabic
45. * document and assigning a score to each sentence. Sentences that contain words
46. * used frequently in the document are given a higher score. You can then choose a
47. * percentage of the highest-scoring sentences to display in the summary.
48. * "ArAutoSummarize" class works best on well-structured documents such as reports,
49. * articles, and scientific papers.
50. *
51. * "ArAutoSummarize" class cuts wordy copy to the bone by counting words and ranking
52. * sentences. First, "ArAutoSummarize" class identifies the most common words in the
53. * document and assigns a "score" to each word--the more frequently a word is used,
54. * the higher the score.
55. *
56. * Then, it "averages" each sentence by adding the scores of its words and dividing
57. * the sum by the number of words in the sentence--the higher the average, the
58. * higher the rank of the sentence. "ArAutoSummarize" class can summarize texts to
59. * specific number of sentences or percentage of the original copy.
60. *
61. * We use statistical approach, with some attention apparently paid to:
62. *
63. * - Location: leading sentences of paragraph, title, introduction, and conclusion.
64. * - Fixed phrases: in-text summaries.
65. * - Frequencies of words, phrases, proper names
66. * - Contextual material: query, title, headline, initial paragraph
67. *
68. * The motivation for this class is the range of applications for key phrases:
69. *
70. * - Mini-summary: Automatic key phrase extraction can provide a quick mini-summary
71. * for a long document. For example, it could be a feature in a web sites; just
72. * click the summarize button when browsing a long web page.
73. *
74. * - Highlights: It can highlight key phrases in a long document, to facilitate
75. * skimming the document.
76. *
77. * - Author Assistance: Automatic key phrase extraction can help an author or editor
78. * who wants to supply a list of key phrases for a document. For example, the
79. * administrator of a web site might want to have a key phrase list at the top of
80. * each web page. The automatically extracted phrases can be a starting point for
81. * further manual refinement by the author or editor.
82. *
83. * - Text Compression: On a device with limited display capacity or limited
84. * bandwidth, key phrases can be a substitute for the full text. For example, an
85. * email message could be reduced to a set of key phrases for display on a pager;
86. * a web page could be reduced for display on a portable wireless web browser.
87. *
88. * This list is not intended to be exhaustive, and there may be some overlap in
89. * the items.
90. *
91. * Example:
92. * <code>
93. * include('./I18N/Arabic.php');
94. * $obj = new I18N_Arabic('AutoSummarize');
95. *
96. * $file = 'Examples/Articles/Ajax.txt';
97. * $r = 20;
98. *
99. * // get contents of a file into a string
100. * $fhandle = fopen($file, "r");
101. * $c = fread($fhandle, filesize($file));
102. * fclose($fhandle);
103. *
104. * $k = $obj->getMetaKeywords($c, $r);
105. * echo '<b><font color=#FFFF00>';
106. * echo 'Keywords:</font></b>';
107. * echo '<p dir="rtl" align="justify">';
108. * echo $k . '</p>';
109. *
110. * $s = $obj->doRateSummarize($c, $r);
111. * echo '<b><font color=#FFFF00>';
112. * echo 'Summary:</font></b>';
113. * echo '<p dir="rtl" align="justify">';
114. * echo $s . '</p>';
115. *
116. * echo '<b><font color=#FFFF00>';
117. * echo 'Full Text:</font></b>';
118. * echo '<p><a class=ar_link target=_blank ';
119. * echo 'href='.$file.'>Source File</a></p>';
120. * </code>
121. *
122. * @category I18N
123. * @package I18N_Arabic
124. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
125. * @copyright 2006-2016 Khaled Al-Sham'aa
126. *
127. * @license LGPL <http://www.gnu.org/licenses/lgpl.txt>
128. * @link http://www.ar-php.org
129. */
131. /**
132. * This PHP class do automatic keyphrase extraction to provide a quick
133. * mini-summary for a long Arabic document
134. *
135. * @category I18N
136. * @package I18N_Arabic
137. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
138. * @copyright 2006-2016 Khaled Al-Sham'aa
139. *
140. * @license LGPL <http://www.gnu.org/licenses/lgpl.txt>
141. * @link http://www.ar-php.org
142. */
143. class I18N_Arabic_AutoSummarize
144. {
145. private $_normalizeAlef = array('أ','إ','آ');
146. private $_normalizeDiacritics = array('َ','ً','ُ','ٌ','ِ','ٍ','ْ','ّ');
148. private $_commonChars = array('ة','ه','ي','ن','و','ت','ل','ا','س','م',
149. 'e', 't', 'a', 'o', 'i', 'n', 's');
151. private $_separators = array('.',"\n",'،','؛','(','[','{',')',']','}',',',';');
153. private $_commonWords = array();
154. private $_importantWords = array();
156. /**
157. * Loads initialize values
158. *
159. * @ignore
160. */
161. public function __construct()
162. {
163. // This common words used in cleanCommon method
164. $words = file(dirname(__FILE__).'/data/ar-stopwords.txt');
165. $en_words = file(dirname(__FILE__).'/data/en-stopwords.txt');
167. $words = array_merge($words, $en_words);
168. $words = array_map('trim', $words);
170. $this->_commonWords = $words;
172. // This important words used in rankSentences method
173. $words = file(dirname(__FILE__).'/data/important-words.txt');
174. $words = array_map('trim', $words);
176. $this->_importantWords = $words;
177. }
179. /**
180. * Load enhanced Arabic stop words list
181. *
182. * @return void
183. */
184. public function loadExtra()
185. {
186. $extra_words = file(dirname(__FILE__).'/data/ar-extra-stopwords.txt');
187. $extra_words = array_map('trim', $extra_words);
189. $this->_commonWords = array_merge($this->_commonWords, $extra_words);
190. }
192. /**
193. * Core summarize function that implement required steps in the algorithm
194. *
195. * @param string $str Input Arabic document as a string
196. * @param string $keywords List of keywords higlited by search process
197. * @param integer $int Sentences value (see $mode effect also)
198. * @param string $mode Mode of sentences count [number|rate]
199. * @param string $output Output mode [summary|highlight]
200. * @param string $style Name of the CSS class you would like to apply
201. *
202. * @return string Output summary requested
203. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
204. */
205. protected function summarize($str, $keywords, $int, $mode, $output, $style=null)
206. {
207. preg_match_all(
208. "/[^\.\n\،\؛\,\;](.+?)[\.\n\،\؛\,\;]/u",
209. $str,
210. $sentences
211. );
212. $_sentences = $sentences[0];
214. if ($mode == 'rate') {
215. $str = preg_replace("/\s{2,}/u", ' ', $str);
216. $totalChars = mb_strlen($str);
217. $totalSentences = count($_sentences);
219. $maxChars = round($int * $totalChars / 100);
220. $int = round($int * $totalSentences / 100);
221. } else {
222. $maxChars = 99999;
223. }
225. $summary = '';
227. $str = strip_tags($str);
228. $normalizedStr = $this->doNormalize($str);
229. $cleanedStr = $this->cleanCommon($normalizedStr);
230. $stemStr = $this->draftStem($cleanedStr);
232. preg_match_all(
233. "/[^\.\n\،\؛\,\;](.+?)[\.\n\،\؛\,\;]/u",
234. $stemStr,
235. $sentences
236. );
237. $_stemmedSentences = $sentences[0];
239. $wordRanks = $this->rankWords($stemStr);
241. if ($keywords) {
242. $keywords = $this->doNormalize($keywords);
243. $keywords = $this->draftStem($keywords);
244. $words = explode(' ', $keywords);
246. foreach ($words as $word) {
247. $wordRanks[$word] = 1000;
248. }
249. }
251. $sentencesRanks = $this->rankSentences(
252. $_sentences,
253. $_stemmedSentences,
254. $wordRanks
255. );
257. list($sentences, $ranks) = $sentencesRanks;
259. $minRank = $this->minAcceptedRank($sentences, $ranks, $int, $maxChars);
261. $totalSentences = count($ranks);
263. for ($i = 0; $i < $totalSentences; $i++) {
264. if ($sentencesRanks[1][$i] >= $minRank) {
265. if ($output == 'summary') {
266. $summary .= ' '.$sentencesRanks[0][$i];
267. } else {
268. $summary .= '<span class="' . $style .'">' .
269. $sentencesRanks[0][$i] . '</span>';
270. }
271. } else {
272. if ($output == 'highlight') {
273. $summary .= $sentencesRanks[0][$i];
274. }
275. }
276. }
278. if ($output == 'highlight') {
279. $summary = str_replace("\n", '<br />', $summary);
280. }
282. return $summary;
283. }
285. /**
286. * Summarize input Arabic string (document content) into specific number of
287. * sentences in the output
288. *
289. * @param string $str Input Arabic document as a string
290. * @param integer $int Number of sentences required in output summary
291. * @param string $keywords List of keywords higlited by search process
292. *
293. * @return string Output summary requested
294. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
295. */
296. public function doSummarize($str, $int, $keywords)
297. {
298. $summary = $this->summarize(
299. $str, $keywords, $int, 'number', 'summary', $style
300. );
302. return $summary;
303. }
305. /**
306. * Summarize percentage of the input Arabic string (document content) into output
307. *
308. * @param string $str Input Arabic document as a string
309. * @param integer $rate Rate of output summary sentence number as
310. * percentage of the input Arabic string
311. * (document content)
312. * @param string $keywords List of keywords higlited by search process
313. *
314. * @return string Output summary requested
315. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
316. */
317. public function doRateSummarize($str, $rate, $keywords)
318. {
319. $summary = $this->summarize(
320. $str, $keywords, $rate, 'rate', 'summary', $style
321. );
323. return $summary;
324. }
326. /**
327. * Highlight key sentences (summary) of the input string (document content)
328. * using CSS and send the result back as an output
329. *
330. * @param string $str Input Arabic document as a string
331. * @param integer $int Number of key sentences required to be
332. * highlighted in the input string
333. * (document content)
334. * @param string $keywords List of keywords higlited by search process
335. * @param string $style Name of the CSS class you would like to apply
336. *
337. * @return string Output highlighted key sentences summary (using CSS)
338. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
339. */
340. public function highlightSummary($str, $int, $keywords, $style)
341. {
342. $summary = $this->summarize(
343. $str, $keywords, $int, 'number', 'highlight', $style
344. );
346. return $summary;
347. }
349. /**
350. * Highlight key sentences (summary) as percentage of the input string
351. * (document content) using CSS and send the result back as an output.
352. *
353. * @param string $str Input Arabic document as a string
354. * @param integer $rate Rate of highlighted key sentences summary
355. * number as percentage of the input Arabic
356. * string (document content)
357. * @param string $keywords List of keywords higlited by search process
358. * @param string $style Name of the CSS class you would like to apply
359. *
360. * @return string Output highlighted key sentences summary (using CSS)
361. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
362. */
363. public function highlightRateSummary($str, $rate, $keywords, $style)
364. {
365. $summary = $this->summarize(
366. $str, $keywords, $rate, 'rate', 'highlight', $style
367. );
369. return $summary;
370. }
372. /**
373. * Extract keywords from a given Arabic string (document content)
374. *
375. * @param string $str Input Arabic document as a string
376. * @param integer $int Number of keywords required to be extracting
377. * from input string (document content)
378. *
379. * @return string List of the keywords extracting from input Arabic string
380. * (document content)
381. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
382. */
383. public function getMetaKeywords($str, $int)
384. {
385. $patterns = array();
386. $replacements = array();
387. $metaKeywords = '';
389. array_push($patterns, '/\.|\n|\،|\؛|\(|\[|\{|\)|\]|\}|\,|\;/u');
390. array_push($replacements, ' ');
391. $str = preg_replace($patterns, $replacements, $str);
393. $normalizedStr = $this->doNormalize($str);
394. $cleanedStr = $this->cleanCommon($normalizedStr);
396. $str = preg_replace('/(\W)ال(\w{3,})/u', '\\1\\2', $cleanedStr);
397. $str = preg_replace('/(\W)وال(\w{3,})/u', '\\1\\2', $str);
398. $str = preg_replace('/(\w{3,})هما(\W)/u', '\\1\\2', $str);
399. $str = preg_replace('/(\w{3,})كما(\W)/u', '\\1\\2', $str);
400. $str = preg_replace('/(\w{3,})تين(\W)/u', '\\1\\2', $str);
401. $str = preg_replace('/(\w{3,})هم(\W)/u', '\\1\\2', $str);
402. $str = preg_replace('/(\w{3,})هن(\W)/u', '\\1\\2', $str);
403. $str = preg_replace('/(\w{3,})ها(\W)/u', '\\1\\2', $str);
404. $str = preg_replace('/(\w{3,})نا(\W)/u', '\\1\\2', $str);
405. $str = preg_replace('/(\w{3,})ني(\W)/u', '\\1\\2', $str);
406. $str = preg_replace('/(\w{3,})كم(\W)/u', '\\1\\2', $str);
407. $str = preg_replace('/(\w{3,})تم(\W)/u', '\\1\\2', $str);
408. $str = preg_replace('/(\w{3,})كن(\W)/u', '\\1\\2', $str);
409. $str = preg_replace('/(\w{3,})ات(\W)/u', '\\1\\2', $str);
410. $str = preg_replace('/(\w{3,})ين(\W)/u', '\\1\\2', $str);
411. $str = preg_replace('/(\w{3,})تن(\W)/u', '\\1\\2', $str);
412. $str = preg_replace('/(\w{3,})ون(\W)/u', '\\1\\2', $str);
413. $str = preg_replace('/(\w{3,})ان(\W)/u', '\\1\\2', $str);
414. $str = preg_replace('/(\w{3,})تا(\W)/u', '\\1\\2', $str);
415. $str = preg_replace('/(\w{3,})وا(\W)/u', '\\1\\2', $str);
416. $str = preg_replace('/(\w{3,})ة(\W)/u', '\\1\\2', $str);
418. $stemStr = preg_replace('/(\W)\w{1,3}(\W)/u', '\\2', $str);
420. $wordRanks = $this->rankWords($stemStr);
422. arsort($wordRanks, SORT_NUMERIC);
424. $i = 1;
425. foreach ($wordRanks as $key => $value) {
426. if ($this->acceptedWord($key)) {
427. $metaKeywords .= $key . '، ';
428. $i++;
429. }
430. if ($i > $int) {
431. break;
432. }
433. }
435. $metaKeywords = mb_substr($metaKeywords, 0, -2);
437. return $metaKeywords;
438. }
440. /**
441. * Normalized Arabic document
442. *
443. * @param string $str Input Arabic document as a string
444. *
445. * @return string Normalized Arabic document
446. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
447. */
448. protected function doNormalize($str)
449. {
450. $str = str_replace($this->_normalizeAlef, 'ا', $str);
451. $str = str_replace($this->_normalizeDiacritics, '', $str);
452. $str = strtr(
453. $str,
454. 'ABCDEFGHIJKLMNOPQRSTUVWXYZ',
455. 'abcdefghijklmnopqrstuvwxyz'
456. );
458. return $str;
459. }
461. /**
462. * Extracting common Arabic words (roughly)
463. * from input Arabic string (document content)
464. *
465. * @param string $str Input normalized Arabic document as a string
466. *
467. * @return string Arabic document as a string free of common words (roughly)
468. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
469. */
470. public function cleanCommon($str)
471. {
472. $str = str_replace($this->_commonWords, ' ', $str);
474. return $str;
475. }
477. /**
478. * Remove less significant Arabic letter from given string (document content).
479. * Please note that output will not be human readable.
480. *
481. * @param string $str Input Arabic document as a string
482. *
483. * @return string Output string after removing less significant Arabic letter
484. * (not human readable output)
485. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
486. */
487. protected function draftStem($str)
488. {
489. $str = str_replace($this->_commonChars, '', $str);
490. return $str;
491. }
493. /**
494. * Ranks words in a given Arabic string (document content). That rank refers
495. * to the frequency of that word appears in that given document.
496. *
497. * @param string $str Input Arabic document as a string
498. *
499. * @return hash Associated array where document words referred by index and
500. * those words ranks referred by values of those array items.
501. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
502. */
503. protected function rankWords($str)
504. {
505. $wordsRanks = array();
507. $str = str_replace($this->_separators, ' ', $str);
508. $words = preg_split("/[\s,]+/u", $str);
510. foreach ($words as $word) {
511. if (isset($wordsRanks[$word])) {
512. $wordsRanks[$word]++;
513. } else {
514. $wordsRanks[$word] = 1;
515. }
516. }
518. foreach ($wordsRanks as $wordRank => $total) {
519. if (mb_substr($wordRank, 0, 1) == 'و') {
520. $subWordRank = mb_substr($wordRank, 1, mb_strlen($wordRank) - 1);
521. if (isset($wordsRanks[$subWordRank])) {
522. unset($wordsRanks[$wordRank]);
523. $wordsRanks[$subWordRank] += $total;
524. }
525. }
526. }
528. return $wordsRanks;
529. }
531. /**
532. * Ranks sentences in a given Arabic string (document content).
533. *
534. * @param array $sentences Sentences of the input Arabic document
535. * as an array
536. * @param array $stemmedSentences Stemmed sentences of the input Arabic
537. * document as an array
538. * @param array $arr Words ranks array (word as an index and
539. * value refer to the word frequency)
540. *
541. * @return array Two dimension array, first item is an array of document
542. * sentences, second item is an array of ranks of document
543. * sentences.
544. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
545. */
546. protected function rankSentences($sentences, $stemmedSentences, $arr)
547. {
548. $sentenceArr = array();
549. $rankArr = array();
551. $max = count($sentences);
553. for ($i = 0; $i < $max; $i++) {
554. $sentence = $sentences[$i];
556. $w = 0;
557. $first = mb_substr($sentence, 0, 1);
558. $last = mb_substr($sentence, -1, 1);
560. if ($first == "\n") {
561. $w += 3;
562. } elseif (in_array($first, $this->_separators)) {
563. $w += 2;
564. } else {
565. $w += 1;
566. }
568. if ($last == "\n") {
569. $w += 3;
570. } elseif (in_array($last, $this->_separators)) {
571. $w += 2;
572. } else {
573. $w += 1;
574. }
576. foreach ($this->_importantWords as $word) {
577. if ($word != '') {
578. $w += mb_substr_count($sentence, $word);
579. }
580. }
582. $sentence = mb_substr(mb_substr($sentence, 0, -1), 1);
583. if (!in_array($first, $this->_separators)) {
584. $sentence = $first . $sentence;
585. }
587. $stemStr = $stemmedSentences[$i];
588. $stemStr = mb_substr($stemStr, 0, -1);
590. $words = preg_split("/[\s,]+/u", $stemStr);
592. $totalWords = count($words);
593. if ($totalWords > 4) {
594. $totalWordsRank = 0;
596. foreach ($words as $word) {
597. if (isset($arr[$word])) {
598. $totalWordsRank += $arr[$word];
599. }
600. }
602. $wordsRank = $totalWordsRank / $totalWords;
603. $sentenceRanks = $w * $wordsRank;
605. array_push($sentenceArr, $sentence . $last);
606. array_push($rankArr, $sentenceRanks);
607. }
608. }
610. $sentencesRanks = array($sentenceArr, $rankArr);
612. return $sentencesRanks;
613. }
615. /**
616. * Calculate minimum rank for sentences which will be including in the summary
617. *
618. * @param array $str Document sentences
619. * @param array $arr Sentences ranks
620. * @param integer $int Number of sentences you need to include in your summary
621. * @param integer $max Maximum number of characters accepted in your summary
622. *
623. * @return integer Minimum accepted sentence rank (sentences with rank more
624. * than this will be listed in the document summary)
625. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
626. */
627. protected function minAcceptedRank($str, $arr, $int, $max)
628. {
629. $len = array();
631. foreach ($str as $line) {
632. $len[] = mb_strlen($line);
633. }
635. rsort($arr, SORT_NUMERIC);
637. $totalChars = 0;
639. for ($i=0; $i<=$int; $i++) {
641. if (!isset($arr[$i])) {
642. $minRank = 0;
643. break;
644. }
646. $totalChars += $len[$i];
648. if ($totalChars >= $max) {
649. $minRank = $arr[$i];
650. break;
651. }
653. $minRank = $arr[$i];
654. }
656. return $minRank;
657. }
659. /**
660. * Check some conditions to know if a given string is a formal valid word or not
661. *
662. * @param string $word String to be checked if it is a valid word or not
663. *
664. * @return boolean True if passed string is accepted as a valid word else
665. * it will return False
666. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
667. */
668. protected function acceptedWord($word)
669. {
670. $accept = true;
672. if (mb_strlen($word) < 3) {
673. $accept = false;
674. }
676. return $accept;
677. }
678. }

Documentation generated on Fri, 01 Jan 2016 10:25:52 +0200 by phpDocumentor 1.4.0