Search Module won't index special characters

zwelldesigns

Hi!

I have special characters with diacritics and in alternate writing systems (Devanagari) in my pages. I've installed the Search Module and when it crawls my pages, it does not index any special characters. For instance, on a page that has this content:
कईं पाँब · kaĩĩ pããba

It will index it with all special characters removed:
ka pba

It seems that this probably originates from the scrape_page or index_page functions in the Search module's Fuel_search library, but I cannot really tell where it comes from internal to those functions. Is there a way to fix this?

Thanks!

admin

I would try debugging in the Fuel_search library class to see if the characters are getting converted somehow. I would probably check the page_xpath() function (line 720) and see what the value of the content is after the convert encoding.

zwelldesigns

Thanks for that hint, I figured it out! clean() on line 1116 of the Fuel_search library was calling $cleaning_funcs from the search config file. Apparently zap_gremlins on line 85 of the search config file was causing the issue (I have discovered that someone else had this issue https://forum.getfuelcms.com/discussion/2007/search-module-problems-with-indexing). I removed it and it works now!

Will removing zap_gremlins cause any other issues?
Thanks!