Solr search in wcs

Objective:

Through following steps, we will:

a.      Use WCS search preprocess to create additional temporary tables to hold the flatten attributes.

b.      Update Solr schema to support new fields

c.      Update DIH configuration SQL to fetch new data from temporary table.

d.      Extended search profile to allow new Solr fields to be in Solr search result

e.      Update jsp to use new search profile.

Procedure:

2.      Configuration in search preprocess:

Add new configuration file in directory: WC_installdir /instances/ instance_name /search\pre-processConfig/MC_ masterCatalogId / databaseType,

For example: search/MC_10001\DB2

a.      Name: wc-dataimport-preprocess-<your own name>.xml; keep the prefix as “wc-dataimport-preprocess”.  (A little trick is that you can name your own file name starting with “a”, for example, wc-dataimport-preprocess-a-mynewdata-attribute.xml ,so the pre-process shell script will run it first by alphabetical order, this will help in development time to verify the testing result without waiting for the whole process completes.)

b.      Add query configuration to populate temporary database table for data import

c.      Follow the instruction to run di-preprocess utility, exam the new temporary table and its data content.

Structure of data-processing-config element:

   

<_config:data-processing-configprocessor="com.ibm.commerce.foundation.dataimport.preprocess.StaticAttributeDataPreProcessor"masterCatalogId="10001" fetchSize="500" batchSize="500">                <_config:table definition="CREATE TABLE TI_MY_TABLE_#lang_tag# (CATENTRY_ID BIGINT NOT NULL, NEWDATA VARCHAR(256) ,PRIMARY KEY (CATENTRY_ID))" name=" TI_MY_TABLE_#lang_tag#"/>                <_config:query sql="SELECT NEWCOLUMN AS NEWDATA, CATENTRY_ID FROM MY_OWN_TABLE"/>                <_config:mapping>                  <_config:key queryColumn="CATENTRY_ID" tableColumn="CATENTRY_ID"/>                  <_config:column-mapping>                        <_config:column-column-mapping>                                                           <_config:column-column queryColumn="NEWDATA"tableColumn="NEWDATA"/>                        </_config:column-column-mapping>                        </_config:column-mapping>                </_config:mapping>             </_config:data-processing-config>

For the simple table creation, the StaticAttributeDataPreProcessor is enough. It will translate your “language_id” and #lang_tag# parameters. If you need more control on the table creation, create your own PreProcessor by extending AbstractDataPreProcessor Create your table definition with dynamic language tag. It will be translated into the actual langId. The negative sign will be trimmed off. The simplified query to select the dataset. The resuleset data will be inserted into above table. Map the primary key column names. Map the returned SELECT resultset column name to temporary table column names.  It is better to keep them the same.

              

2.      Adding new Solr fields into Solr schema

(You can learn the details in Apache Solr about this)

Update schema.xml in each Solr core, for example: \solr\home\MC_10001\en_US\CatalogEntry\conf\schema.xml

Add below:

<field name="newField" type="string" indexed="true" stored="true"multiValued="true"/>

Where:

           Type- depends on your search needs, you may need use different data type for this. For example, you need consider whether this should be tokenized, case sensitive, etc. You can follow WCS naming convention.

           Indexed – only indexed field can be searchable in Solr. If you don’t need the newField to be searchable, as it maybe just some additional descriptive field, set it to false.

           Stored – if true, you can retrieve its value through Solr request. You cann’t see its value if you set it false. It could become useful when you only want user to search by it, but show other fields instead other than showing this field.

           multivalued – whether you want to set this value as an array type to hold multiple values.

If you want this newField being part of your default search field, do this :

<copyField source="newField" dest="defaultSearch"/>

3.      DIH (dataimport handler) configuration

(You can learn the details in Apache Solr about this)

Update wc-data-config.xml in  \solr\home\MC_10001\en_US\CatalogEntry\conf

a.      Update query  to join the new temporary table

·        Add additional line in SELECT clause to return the newData

TI_MY_TABLE.NEWDATA as MY_NEWDATA

·        Add left outer join line in FROM clause (use alias name without langId)

LEFT OUTER JOIN TI_MY_TABLE_1 TI_MY_TABLE ON ( CATENTRY.CATENTRY_ID= TI_MY_TABLE.CATENTRY_ID)

b.      Add below:

<field  column="MY_NEWDATA" name="newData"/>        

If it is multi value Solr field, you have to create the data with separator, like “;”. Then use splitBy to split it.

For example:

           <field  sourceColName="MY_NEWDATA" column="newData" splitBy=";"/>        

Note1: in case of <field  column="MY_NEWDATA" name="newData"/>, “column” is database column name.

             In <field  sourceColName="MY_NEWDATA" column="newData" splitBy=";"/>   , “column” is schema field name. ( A design naming convention needs to be revised by Solr.)

Note2: if you use splitBy, better to use different sourceColName and column names. Or same name in case sensitive format.

           The format of <field  sourceColName="NEWDATA" column="newData" splitBy=";"/>     will have issue in the final result. The unsplit data will be shown.

c.      Repeat above steps for other cores.

4.      Create a new search profile “My_own_search_profile”

a.      In wc-search.xml at Search\xml\config\com.ibm.commerce.catalog-ext,

if you donot have “-ext” directory, you need create one. Copy the header part from existing wc-search.xml.

add below, assuming this will extends IBM_findProductByIds_Summary:

<_config:profile extends="IBM_findProductByIds_Summary" name="My_own_search_profile ">

        <_config:result inherits="true" >

            <_config:field name="newData"/>

        </_config:result>

    </_config:profile>

5.      Update JSP

a.      In JSP file, set the above new search profile to WCF:REST request.  

6.      Search server setup.

In order to allow WCS Search server to accept new search profile, you need add the new search file into the Search-Rest configuration.

Add new profiles My_own_search_profile in searchProfile list of wc-rest-resourceconfig.xml in Search-Rest\WebContent\WEB-INF\config\com.ibm.commerce.rest-ext

    If you donot have –ext, create one, and copy wc-rest-resourceconfig.xml from com.ibm.commerce.rest directory .

Assuming you are using path “store/{storeId}/productview/byIds”:

<GetUri uri="store/{storeId}/productview/byIds"

description="Get products by unique IDs"

searchProfile=" My_own_search_profile,<existing list…..>”

This will allow search server to accept new searchProfile.