I18N_Arabic

# Source for file Query.php

Documentation is available at Query.php

1. <?php
2. /**
3. * ----------------------------------------------------------------------
4. *
5. * Copyright (c) 2006-2016 Khaled Al-Sham'aa.
6. *
7. * http://www.ar-php.org
8. *
9. * PHP Version 5
10. *
11. * ----------------------------------------------------------------------
12. *
13. * LICENSE
14. *
15. * This program is open source product; you can redistribute it and/or
16. * modify it under the terms of the GNU Lesser General Public License (LGPL)
17. * as published by the Free Software Foundation; either version 3
18. * of the License, or (at your option) any later version.
19. *
20. * This program is distributed in the hope that it will be useful,
21. * but WITHOUT ANY WARRANTY; without even the implied warranty of
22. * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
23. * GNU Lesser General Public License for more details.
24. *
25. * You should have received a copy of the GNU Lesser General Public License
26. * along with this program. If not, see <http://www.gnu.org/licenses/lgpl.txt>.
27. *
28. * ----------------------------------------------------------------------
29. *
30. * Class Name: Arabic Queary Class
31. *
32. * Filename: Query.php
33. *
34. * Original Author(s): Khaled Al-Sham'aa <khaled@ar-php.org>
35. *
36. * Purpose: Build WHERE condition for SQL statement using MySQL REGEXP and
37. * Arabic lexical rules
38. *
39. * ----------------------------------------------------------------------
40. *
41. * Arabic Queary Class
42. *
43. * PHP class build WHERE condition for SQL statement using MySQL REGEXP and
44. * Arabic lexical rules.
45. *
46. * With the exception of the Qur'an and pedagogical texts, Arabic is generally
47. * written without vowels or other graphic symbols that indicate how a word is
48. * pronounced. The reader is expected to fill these in from context. Some of the
49. * graphic symbols include sukuun, which is placed over a consonant to indicate that
50. * it is not followed by a vowel; shadda, written over a consonant to indicate it is
51. * doubled; and hamza, the sign of the glottal stop, which can be written above or
52. * below (alif) at the beginning of a word, or on (alif), (waaw), (yaa'),
53. * or by itself on the line elsewhere. Also, common spelling differences regularly
54. * appear, including the use of (haa') for (taa' marbuuta) and (alif maqsuura)
55. * for (yaa'). These features of written Arabic, which are also seen in Hebrew as
56. * well as other languages written with Arabic script (such as Farsi, Pashto, and
57. * Urdu), make analyzing and searching texts quite challenging. In addition, Arabic
58. * morphology and grammar are quite rich and present some unique issues for
59. * information retrieval applications.
60. *
61. * There are essentially three ways to search an Arabic text with Arabic queries:
62. * literal, stem-based or root-based.
63. *
64. * A literal search, the simplest search and retrieval method, matches documents
65. * based on the search terms exactly as the user entered them. The advantage of this
66. * technique is that the documents returned will without a doubt contain the exact
67. * term for which the user is looking. But this advantage is also the biggest
68. * disadvantage: many, if not most, of the documents containing the terms in
69. * different forms will be missed. Given the many ambiguities of written Arabic, the
70. * success rate of this method is quite low. For example, if the user searches
71. * for (kitaab, book), he or she will not find documents that only
72. * contain (`al-kitaabu, the book).
73. *
74. * Stem-based searching, a more complicated method, requires some normalization of
75. * the original texts and the queries. This is done by removing the vowel signs,
76. * unifying the hamza forms and removing or standardizing the other signs.
77. * Additionally, grammatical affixes and other constructions which attach directly
78. * to words, such as conjunctions, prepositions, and the definite article, should be
79. * identified and removed. Finally, regular and irregular plural forms need to be
80. * identified and reduced to their singular forms. Performing this type of stemming
81. * leads to more successful searches, but can be problematic due to over-generation
82. * or incorrect generation of stems.
83. *
84. * A third method for searching Arabic texts is to index and search for the root
85. * forms of each word. Since most verbs and nouns in Arabic are derived from
86. * triliteral (or, rarely, quadriliteral) roots, identifying the underlying root of
87. * each word theoretically retrieves most of the documents containing a given search
88. * term regardless of form. However, there are some significant challenges with this
89. * approach. Determining the root for a given word is extremely difficult, since it
90. * requires a detailed morphological, syntactic and semantic analysis of the text to
91. * fully disambiguate the root forms. The issue is complicated further by the fact
92. * that not all words are derived from roots. For example, loan words (words
93. * borrowed from another language) are not based on root forms, although there are
94. * even exceptions to this rule. For example, some loans that have a structure
95. * similar to triliteral roots, such as the English word film, are handled
96. * grammatically as if they were root-based, adding to the complexity of this type
97. * of search. Finally, the root can serve as the foundation for a wide variety of
98. * words with related meanings. The root (k-t-b) is used for many words related
99. * to writing, including (kataba, to write), (kitaab, book), (maktab,
100. * office), and (kaatib, author). But the same root is also used for regiment/
101. * battalion, (katiiba). As a result, searching based on root forms results in
102. * very high recall, but precision is usually quite low.
103. *
104. * While search and retrieval of Arabic text will never be an easy task, relying on
105. * linguistic analysis tools and methods can help make the process more successful.
106. * Ultimately, the search method you choose should depend on how critical it is to
107. * retrieve every conceivable instance of a word or phrase and the resources you
108. * have to process search returns in order to determine their true relevance.
109. *
110. * Source: Volume 13 Issue 7 of MultiLingual Computing &
111. * Technology published by MultiLingual Computing, Inc., 319 North First Ave.,
112. * Sandpoint, Idaho, USA, 208-263-8178, Fax: 208-263-6310.
113. *
114. * Example:
115. * <code>
116. * include('./I18N/Arabic.php');
117. * $obj = new I18N_Arabic('Query');
118. *
119. * $dbuser = 'root';
120. * $dbpwd = '';
121. * $dbname = 'test';
122. *
123. * try {
124. * $dbh = new PDO('mysql:host=localhost;dbname='.$dbname, $dbuser, $dbpwd);
125. *
126. * // Set the error reporting attribute
127. * $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
128. *
129. * $dbh->exec("SET NAMES 'utf8'");
130. *
131. * if ($_GET['keyword'] != '') {
132. * $keyword = @$_GET['keyword'];
133. * $keyword = str_replace('\"', '"', $keyword);
134. *
135. * $obj->setStrFields('headline');
136. * $obj->setMode($_GET['mode']);
137. *
138. * $strCondition = $Arabic->getWhereCondition($keyword);
139. * } else {
140. * $strCondition = '1';
141. * }
142. *
143. * $StrSQL = "SELECT `headline` FROM `aljazeera` WHERE $strCondition";
144. *
145. * $i = 0;
146. * foreach ($dbh->query($StrSQL) as $row) {
147. * $headline = $row['headline'];
148. * $i++;
149. * if ($i % 2 == 0) {
150. * $bg = "#f0f0f0";
151. * } else {
152. * $bg = "#ffffff";
153. * }
154. * echo "<tr bgcolor=\"$bg\"><td>$headline</td></tr>";
155. * }
156. *
157. * // Close the databse connection
158. * $dbh = null;
159. *
160. * } catch (PDOException $e) {
161. * echo $e->getMessage();
162. * }
163. * </code>
164. *
165. * @category I18N
166. * @package I18N_Arabic
167. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
168. * @copyright 2006-2016 Khaled Al-Sham'aa
169. *
170. * @license LGPL <http://www.gnu.org/licenses/lgpl.txt>
171. * @link http://www.ar-php.org
172. */
174. /**
175. * This PHP class build WHERE condition for SQL statement using MySQL REGEXP and
176. * Arabic lexical rules
177. *
178. * @category I18N
179. * @package I18N_Arabic
180. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
181. * @copyright 2006-2016 Khaled Al-Sham'aa
182. *
183. * @license LGPL <http://www.gnu.org/licenses/lgpl.txt>
184. * @link http://www.ar-php.org
185. */
186. class I18N_Arabic_Query
187. {
188. private $_fields = array();
189. private $_lexPatterns = array();
190. private $_lexReplacements = array();
191. private $_mode = 0;
193. /**
194. * Loads initialize values
195. */
196. public function __construct()
197. {
198. $xml = simplexml_load_file(dirname(__FILE__).'/data/ArQuery.xml');
200. foreach ($xml->xpath("//preg_replace[@function='__construct']/pair")
201. as $pair) {
203. array_push($this->_lexPatterns, (string)$pair->search);
204. array_push($this->_lexReplacements, (string)$pair->replace);
205. }
206. }
208. /**
209. * Setting value for $_fields array
210. *
211. * @param array $arrConfig Name of the fields that SQL statement will search
212. * them (in array format where items are those
213. * fields names)
214. *
215. * @return object $this to build a fluent interface
216. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
217. */
218. public function setArrFields($arrConfig)
219. {
220. if (is_array($arrConfig)) {
221. // Get _fields array
222. $this->_fields = $arrConfig;
223. }
225. return $this;
226. }
228. /**
229. * Setting value for $_fields array
230. *
231. * @param string $strConfig Name of the fields that SQL statement will search
232. * them (in string format using comma as delimated)
233. *
234. * @return object $this to build a fluent interface
235. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
236. */
237. public function setStrFields($strConfig)
238. {
239. if (is_string($strConfig)) {
240. // Get _fields array
241. $this->_fields = explode(',', $strConfig);
242. }
244. return $this;
245. }
247. /**
248. * Setting $mode propority value that refer to search mode
249. * [0 for OR logic | 1 for AND logic]
250. *
251. * @param integer $mode Setting value to be saved in the $mode propority
252. *
253. * @return object $this to build a fluent interface
254. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
255. */
256. public function setMode($mode)
257. {
258. if (in_array($mode, array('0', '1'))) {
259. // Set search mode [0 for OR logic | 1 for AND logic]
260. $this->_mode = $mode;
261. }
263. return $this;
264. }
266. /**
267. * Getting $mode propority value that refer to search mode
268. * [0 for OR logic | 1 for AND logic]
269. *
270. * @return integer Value of $mode properity
271. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
272. */
273. public function getMode()
274. {
275. // Get search mode value [0 for OR logic | 1 for AND logic]
276. return $this->_mode;
277. }
279. /**
280. * Getting values of $_fields Array in array format
281. *
282. * @return array Value of $_fields array in Array format
283. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
284. */
285. public function getArrFields()
286. {
287. $fields = $this->_fields;
289. return $fields;
290. }
292. /**
293. * Getting values of $_fields array in String format (comma delimated)
294. *
295. * @return string Values of $_fields array in String format (comma delimated)
296. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
297. */
298. public function getStrFields()
299. {
300. $fields = implode(',', $this->_fields);
302. return $fields;
303. }
305. /**
306. * Build WHERE section of the SQL statement using defind lex's rules, search
307. * mode [AND | OR], and handle also phrases (inclosed by "") using normal
308. * LIKE condition to match it as it is.
309. *
310. * @param string $arg String that user search for in the database table
311. *
312. * @return string The WHERE section in SQL statement
313. * (MySQL database engine format)
314. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
315. */
316. public function getWhereCondition($arg)
317. {
318. $sql = '';
320. //$arg = mysql_real_escape_string($arg);
321. $search = array("\\", "\x00", "\n", "\r", "'", '"', "\x1a");
322. $replace = array("\\\\","\\0","\\n", "\\r", "\'", '\"', "\\Z");
323. $arg = str_replace($search, $replace, $arg);
325. // Check if there are phrases in $arg should handle as it is
326. $phrase = explode("\"", $arg);
328. if (count($phrase) > 2) {
329. // Re-init $arg variable
330. // (It will contain the rest of $arg except phrases).
331. $arg = '';
333. for ($i = 0; $i < count($phrase); $i++) {
334. $subPhrase = $phrase[$i];
335. if ($i % 2 == 0 && $subPhrase != '') {
336. // Re-build $arg variable after restricting phrases
337. $arg .= $subPhrase;
338. } elseif ($i % 2 == 1 && $subPhrase != '') {
339. // Handle phrases using reqular LIKE matching in MySQL
340. $this->wordCondition[] = $this->getWordLike($subPhrase);
341. }
342. }
343. }
345. // Handle normal $arg using lex's and regular expresion
346. $words = preg_split('/\s+/', trim($arg));
348. foreach ($words as $word) {
349. //if (is_numeric($word) || strlen($word) > 2) {
350. // Take off all the punctuation
351. //$word = preg_replace("/\p{P}/", '', $word);
352. $exclude = array('(', ')', '[', ']', '{', '}', ',', ';', ':',
353. '?', '!', '،', '؛', '؟');
354. $word = str_replace($exclude, '', $word);
356. $this->wordCondition[] = $this->getWordRegExp($word);
357. //}
358. }
360. if (!empty($this->wordCondition)) {
361. if ($this->_mode == 0) {
362. $sql = '(' . implode(') OR (', $this->wordCondition) . ')';
363. } elseif ($this->_mode == 1) {
364. $sql = '(' . implode(') AND (', $this->wordCondition) . ')';
365. }
366. }
368. return $sql;
369. }
371. /**
372. * Search condition in SQL format for one word in all defind fields using
373. * REGEXP clause and lex's rules
374. *
375. * @param string $arg String (one word) that you want to build a condition for
376. *
377. * @return string sub SQL condition (for internal use)
378. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
379. */
380. protected function getWordRegExp($arg)
381. {
382. $arg = $this->lex($arg);
383. //$sql = implode(" REGEXP '$arg' OR ", $this->_fields) . " REGEXP '$arg'";
384. $sql = ' REPLACE(' .
385. implode(", 'ـ', '') REGEXP '$arg' OR REPLACE(", $this->_fields) .
386. ", 'ـ', '') REGEXP '$arg'";
389. return $sql;
390. }
392. /**
393. * Search condition in SQL format for one word in all defind fields using
394. * normal LIKE clause
395. *
396. * @param string $arg String (one word) that you want to build a condition for
397. *
398. * @return string sub SQL condition (for internal use)
399. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
400. */
401. protected function getWordLike($arg)
402. {
403. $sql = implode(" LIKE '$arg' OR ", $this->_fields) . " LIKE '$arg'";
405. return $sql;
406. }
408. /**
409. * Get more relevant order by section related to the user search keywords
410. *
411. * @param string $arg String that user search for in the database table
412. *
413. * @return string sub SQL ORDER BY section
414. * @author Saleh AlMatrafe <saleh@saleh.cc>
415. */
416. public function getOrderBy($arg)
417. {
418. // Check if there are phrases in $arg should handle as it is
419. $phrase = explode("\"", $arg);
420. if (count($phrase) > 2) {
421. // Re-init $arg variable
422. // (It will contain the rest of $arg except phrases).
423. $arg = '';
424. for ($i = 0; $i < count($phrase); $i++) {
425. if ($i % 2 == 0 && $phrase[$i] != '') {
426. // Re-build $arg variable after restricting phrases
427. $arg .= $phrase[$i];
428. } elseif ($i % 2 == 1 && $phrase[$i] != '') {
429. // Handle phrases using reqular LIKE matching in MySQL
430. $wordOrder[] = $this->getWordLike($phrase[$i]);
431. }
432. }
433. }
435. // Handle normal $arg using lex's and regular expresion
436. $words = explode(' ', $arg);
437. foreach ($words as $word) {
438. if ($word != '') {
439. $wordOrder[] = 'CASE WHEN ' .
440. $this->getWordRegExp($word) .
441. ' THEN 1 ELSE 0 END';
442. }
443. }
445. $order = '((' . implode(') + (', $wordOrder) . ')) DESC';
447. return $order;
448. }
450. /**
451. * This method will implement various regular expressin rules based on
452. * pre-defined Arabic lexical rules
453. *
454. * @param string $arg String of one word user want to search for
455. *
456. * @return string Regular Expression format to be used in MySQL query statement
457. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
458. */
459. protected function lex($arg)
460. {
461. $arg = preg_replace($this->_lexPatterns, $this->_lexReplacements, $arg);
463. return $arg;
464. }
466. /**
467. * Get most possible Arabic lexical forms for a given word
468. *
469. * @param string $word String that user search for
470. *
471. * @return string list of most possible Arabic lexical forms for a given word
472. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
473. */
474. protected function allWordForms($word)
475. {
476. $wordForms = array($word);
478. $postfix1 = array('كم', 'كن', 'نا', 'ها', 'هم', 'هن');
479. $postfix2 = array('ين', 'ون', 'ان', 'ات', 'وا');
481. $len = mb_strlen($word);
483. if (mb_substr($word, 0, 2) == 'ال') {
484. $word = mb_substr($word, 2);
485. }
487. $wordForms[] = $word;
489. $str1 = mb_substr($word, 0, -1);
490. $str2 = mb_substr($word, 0, -2);
491. $str3 = mb_substr($word, 0, -3);
493. $last1 = mb_substr($word, -1);
494. $last2 = mb_substr($word, -2);
495. $last3 = mb_substr($word, -3);
497. if ($len >= 6 && $last3 == 'تين') {
498. $wordForms[] = $str3;
499. $wordForms[] = $str3 . 'ة';
500. $wordForms[] = $word . 'ة';
501. }
503. if ($len >= 6 && ($last3 == 'كما' || $last3 == 'هما')) {
504. $wordForms[] = $str3;
505. $wordForms[] = $str3 . 'كما';
506. $wordForms[] = $str3 . 'هما';
507. }
509. if ($len >= 5 && in_array($last2, $postfix2)) {
510. $wordForms[] = $str2;
511. $wordForms[] = $str2.'ة';
512. $wordForms[] = $str2.'تين';
514. foreach ($postfix2 as $postfix) {
515. $wordForms[] = $str2 . $postfix;
516. }
517. }
519. if ($len >= 5 && in_array($last2, $postfix1)) {
520. $wordForms[] = $str2;
521. $wordForms[] = $str2.'ي';
522. $wordForms[] = $str2.'ك';
523. $wordForms[] = $str2.'كما';
524. $wordForms[] = $str2.'هما';
526. foreach ($postfix1 as $postfix) {
527. $wordForms[] = $str2 . $postfix;
528. }
529. }
531. if ($len >= 5 && $last2 == 'ية') {
532. $wordForms[] = $str1;
533. $wordForms[] = $str2;
534. }
536. if (($len >= 4 && ($last1 == 'ة' || $last1 == 'ه' || $last1 == 'ت'))
537. || ($len >= 5 && $last2 == 'ات')
538. ) {
539. $wordForms[] = $str1;
540. $wordForms[] = $str1 . 'ة';
541. $wordForms[] = $str1 . 'ه';
542. $wordForms[] = $str1 . 'ت';
543. $wordForms[] = $str1 . 'ات';
544. }
546. if ($len >= 4 && $last1 == 'ى') {
547. $wordForms[] = $str1 . 'ا';
548. }
550. $trans = array('أ' => 'ا', 'إ' => 'ا', 'آ' => 'ا');
551. foreach ($wordForms as $word) {
552. $normWord = strtr($word, $trans);
553. if ($normWord != $word) {
554. $wordForms[] = $normWord;
555. }
556. }
558. $wordForms = array_unique($wordForms);
560. return $wordForms;
561. }
563. /**
564. * Get most possible Arabic lexical forms of user search keywords
565. *
566. * @param string $arg String that user search for
567. *
568. * @return string list of most possible Arabic lexical forms for given keywords
569. * @author Khaled Al-Sham'aa <khaled@ar-php.org>
570. */
571. public function allForms($arg)
572. {
573. $wordForms = array();
574. $words = explode(' ', $arg);
576. foreach ($words as $word) {
577. $wordForms = array_merge($wordForms, $this->allWordForms($word));
578. }
580. $str = implode(' ', $wordForms);
582. return $str;
583. }
584. }

Documentation generated on Fri, 01 Jan 2016 10:26:18 +0200 by phpDocumentor 1.4.0