ESP JDBC Connector can't establish a connection to SQL Server database

The Problem
I setup a new ESP Server and install JDBC Connector on it. But it always threw an exception when I trigger the run from command line:

13:07:59,691 ERROR [JDBCConnector] Excpetion thrown in adapter class: com.fastsearch.esp.connectors.jdbc.JDBCAdapter, Could not connect to database: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "RSA premaster secret error".
13:07:59,693 ERROR [JDBCConnector] Caused by: Could not connect to database: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "RSA premaster secret error".
13:07:59,697 ERROR [JDBCConnector] Caused by: The driver could not establish a secure connection to SQL Server by using Secure Sockets Layer (SSL) encryption. Error: "RSA premaster secret error".
13:07:59,698 ERROR [JDBCConnector] Caused by: RSA premaster secret error
13:07:59,699 ERROR [JDBCConnector] Caused by: SunTlsRsaPremasterSecret KeyGenerator not available


But the connector works fine if I trigger the run from Admin UI.

The Reason
I searched this issue in Google for a couple of hours and someone said the JDBC Connector will always use SSL encryption for SQL Server if it can. There is no way to disable it.

The Solution
Someone mentioned that we can install the new Microsoft JDBC Driver version but it is still helpless. Fortunately, I have another server setup with 'Correct' environment. I compared a quite folders, configure files and figured out the real solution.

After changing the java environment settings with following parameters, the JDBC Connector start working for me. Hopefully it still work for you. :)

Variable	Value
Java_Home	C:\Program Files (x86)\Java\jre
ClassPath	.;%JAVA_HOME%\lib\dt.jar;%JAVA_HOME%\lib\tools.jar
Path	%JAVA_HOME%\bin

ESP JDBC Connector numbs and can't continue

Hi guys,

As you guys knew, I'm working with FAST ESP in the past of few months. Today my team member encountered a big issue, which blocks our task, the JDBC Connector does not work. You know what, all of our data need to be crawled by FAST ESP. So without the ESP, our product/code is a bullshit!!! Right now you can imagining how I'm surprised this morning.

OK, I guess you guys don't want to spend so many time to read my bosh. Let's focus this problem.

Problem
If your data comes from any DBs, you may use JDBC Connector to pull out all data from DB to ESP. I have the same scenario. Only one special point is that I'm wanna use standalone mode, which will use connect.bat file under %JDBC Connector Installed Directory%\bin to trigger the process.

This morning, my friends told me that this batch file numbs and can't continue. It stayed at 'Starting publisher thread 0', like this:


The process sleeps there and won't wake up for never. There is no error, no break and no any other information that you can refer. My friends try to change some settings under our configuration file, change some DB settings, reinstall JDBC Connector but it's still the same. If you have the same issue, then you are lucky because I find out the cause and the solution.

Cause
I have to say I'm so lucky that I know there is a software named ProcessMonitor.exe producted by Microsoft, which can be used to monitor all processes, files and registers access. I use this tool to monitor the java.exe process and found there are some interface/properties in some jar files are not found. (I didn't capture any screenshot when I investigate this issue) Actually I know nothing about Java but I heard some complains that someone's software doesn't work after they upgrade their java version.

Another lucky thing that I can resolve this issue is that I have 3 ESP servers that I can use them to compare the environment. When I look at the other two servers, whose JDBC Connector works fine, the java version is 1.6.0_25-b06. The broken server's java version is 1.6.0_29_b11. At that time, I was thinking this issue may be caused by the upgraded java version. In a fact, I'm right!

Solution
Now, you already knew the culprit and you may get the solution. Yes, the solution is revert the java version. I uninstalled the latest version and reinstall the 1.6.0_25_b06. After rebooting the system, the JDBC Connector comes back. Oh! I'm so lucky!

OK. If you faced the same problem with JDBC Connector, I hope this article can give you some help.

Basic concepts in ESP

In my previous article, I talked about How to install FAST ESP in Windows and Hello World! -- Set up a test search in ESP. In this article, I'm gonna talk about some basic concepts in ESP.
-----------------------------------------------------------
Basic Concepts
The most of following concepts are from the product overview.

Concept	Description
Document set description, indexes
Applies algorithms or business rule-based ranking to the results
Data Flow Overview
Module Overview	Talking about a general ideas about all ESP modules
Basic Concepts	Talking about the concepts in ESP
Content	Data that has not submitted to the FAST ESP system
Document	Processed, searchable content are called Document
Collections	Documents are grouped into different collections. Each collection can have its own processed and indexed way (Index Profile). Also, by setting priority for each collection, we can specify the order of document processing.
Search Profile	Define what to search and how the queries and results should be processed and displayed
Document and Document Element	One content will be converted into a document. Each property of the content will be converted into a document element.
Index Schedule, Profile	FAST Search Engine maps the document's elements to fields. Fields are defined document elements that are to be searchable. Fields can be defined by Index Profile. Multiple fields may be grouped into composite fields, allowing a query to be executed on several fields at the same time.
Enterprise Crawler	Use Enterprise Crawler to access content on Web Site(s)
File Traverser	The file traverser scans specified file directories of file servers.
Pushing Content to Search Engine Using Content API	Use the content API directly to push the content to Search Engine.
Query Side	Three ways to query the result: Search API, HTTP-based Query Interface, FAST Web Service Interface
Content Interface	Integrate of application via C++, Java, .NET
Search Interface
Document Processing Interface	inclusion of customer-defined document processors
Query/Result Processing Interface	provides an interface for dynamic linking of custom query and result processors
Administration Interface	supports API integration for system administration and collection configuration
Security Integration	Security Access Module provides document-level security capabilities for integration with your content and portal infrastructure
SDKs	ESP Content SDK, Search SDK, and Application SDK provide various interfacing capabilities.
Web Service Interface	Web services are a collection of standards and protocols that allow computers to communicate across the internet using XML and the ubiquitous HTTP protocol
Document processing is defined per collection
Document Processing Engine, Pipeline, Stage	One search engine contains multiple pipelines, but one collection can only have one pipeline. One pipeline contains multiple stages. One stage performs a particular document processing task. It takes one or more document elements to be input and the resulting output is new or modified elements that may be further processed
Entity Extraction	Entity extraction is detecting, extracting, and normalizing entities from documents
Extract other entities	Two ways to extract other entities: Using Admin UI to specify additional extractor Via a regular expression document processor
Search Engine Clusters	Search Engine instances are grouped into search engine clusters. A search engine cluster is a group of Search Engine instances that share the same index schema, which is provided by an index profile.
Search Columns and Rows	Sets of indexed documents are stored in all search engine instances within a search column to scale data volume. That means each node in a search rows share the same set of indexed documents. When a query is sent to a cluster, it will be sent to all search engine instances within a search row to scale query rate.
Index profile	An index profile is an XML-based configuration file. It’s an index schema that defines the way documents are searchable. It specifies search properties like: Which document elements are to become searchable fields Which document elements are to become fields that are returned as part of a result How to calculate values that are used for sorting and ranking
The relationship between Document Processing, Indexing and Search Engine Clusters
Index Profile Structure
Scope Search	Used for Indexing customer XML content without any knowledge of the DTD/Schema. Indexing a more dynamic field structure using the Scope Search framework.
Relevancy, Data mining
Linguistic processing
Sorting
Rank value calculations
Query context analysis
Navigation
Contextual Insight
Ranking Concept
Quality
Freshness Boosting
WebAnalyzer	The WebAnalyzer is a FAST ESP module that uses links between documents to improve search relevancy
Tools to modify rank for individual documents	Two tools for modifying rank Search Business Center Boost Bulk Tool
3 boost mechanisms	Absolute Query Boost: Specify an absolute ranking position for a document against a specified query. Or exclude displaying a document against a specified query. Relative Query Boost: Ensure a document is always displayed in first xx (a number) result list against a specified query. Relative Document Boost: Ensure a document is always displayed in first xx (a number) result list whatever user submitted.
Proximity Ranking and Matching	The term proximity denotes the degree to which a query and a document match, based on the distance between the query terms within a document. Two types of proximity: Explicit Proximity Implicit Proximity
Field Collapsing	Two kinds of field collapsing Field collapsing which removes collapsed documents Field collapsing which does not remove collapsed document (default)
Boundary Matching
Duplicate Removal	Different ways of detecting and removing duplicate documents. Crawler Duplicate Removal (The FAST Crawler) Dynamic (Result-Side) Duplicate Removal (may be used to detect and remove duplicates across collections, and also enable a more flexible definition of perceived duplicates) Field Collapsing
GEO Search Overview	The Geo Search feature provides capabilities for filtering, sorting and boosting query results based on geographical location.
Query Modifications	Query processing is configured globally and three ways to modify a query in FAST ESP As an automatic rewrite of the query before execution against the index As a suggested rewrite, typically presented as a search tip on the result page A combination of the two above: The query is first executed in its original form. In case of no hits, the query is automatically resubmitted using the automatic rewrite option, and the new result is presented to the user
Query Resubmission	The resubmission is set per query and used to switch to suggested transformation of the user’s query. There are three kinds of query transformation. Modify: Automatically modified. The modified query is executed and the result set is returned Conditional Modify: Automatically modified only if no hits are returned by the executed query Suggest: Never modified. But a suggested transformed query is returned together with the result set.
FAST Query Language (FQL)
Navigator	Navigators provide functionality for drilling down into the query results based on value distribution of one or more individual fields.
Field Navigator
Deep Navigator
Shallow Navigator
Scope Navigator
Contextual Navigator
Field Navigators for Values in Scope Fields
Taxonomy
FAST Classifier
Unsupervised Clustering

Data Flow
In this section, I'm gonna talk about the data flow. The first one is about how the ESP crawl data.

The second one is about how the ESP handle user search.

Hello World! -- Set up a test search in ESP

In the previous article, I already showed you how to setup ESP in windows. In this article, I'd like to show you how to do a test search in ESP. After this article, you will be able to setup File Traverser connector in ESP, create a collection and search the result.
---------------------------------------------------------------------------
Setup File Traverser connector
1. Integrate the connector into ESP. Open your $FASTSEARCH/etc/NodeConf.xml file, and then add the "<proc>connectorcontroller</proc>" to <startorder> node.

2. Add the following code after </global>:
    <!-- Connector Controller -->
    <process name="connectorcontroller" description="Connector Controller">
        <start>
        <executable>connectorcontroller</executable>
        <parameters>-P $PORT</parameters>
        <port base="3150"/>
        </start>
        <outfile>connectorcontroller.scrap</outfile>
    </process>

3. Run $FASTSEARCH/bin/nctrl reloadcfg
4. Run $FASTSEARCH/bin/nctrl start connectorcontroller

5. Now, your File Traverser should be able to see in ESP Data Source. Setup a shared folder, and then put some pdf files into this folder.

Setup your new collection
1. Go to your FAST ESP log in page

2. Enter your user name and password, click Log in button to go to your home page

3. Click ESP Admin GUI from the right side

4. In your Collection Overview tab under FAST ESP administration page, click Create Collection button

5. Enter a new collection Name and Description and then click the Next button.

6. Since we are doing a demo search, there is only one cluster we can use. Keep the webcluster selected, and then click Next button.

7. In the pipeline configuration page, click the drop down button, select the Generic (webcluster) pipeline, click the Add Selected button and then click Next button.

8. In the data source configuration page, click the drop down button, select the File Traverser and then click the add selected button.

9. In the data source setup page, enter the shared folder path, keep the other default settings and then click submit button.

10. Finally, click OK button to finish the setup

11. Now, a new collection has been setup and it starts to crawl the shared folder.

12. After a few seconds, click the Refresh button on the top right corner. You will see there is a Doc found.

13. Click the Search View from the top.

14. In the new search view tab, you will see the built-in search UI. Enter some keywords and then click search button.