Adding an On-Premise HA Search Server

Adding a New Search Node to the Configuration

Here's how to add a new search node to your configuration:

Take all ingress replicators out of the load balancer rotation.
Wait for all ingress replicators to deliver pending activities. To do this, from the command line run: curl http://ir0x.yourdomain.com:29000/logging/metric/listCounters | grep InQueue

This should return something like the following, which indicates that you are replicating to search01.yourdomain.com:30000 and search02.yourdomain.com:30000, and that the InQueue metric is 0.0, which means that all activities have been delivered:

{"key":"counter>com.jivesoftware.service.activity.stream.replicator.ActivityStreamReplicator>search01.yourdomain.com_30000>InQueue","metric":0.0},
{"key":"counter>com.jivesoftware.service.activity.stream.replicator.ActivityStreamReplicator>search02.yourdomain.com_30000>InQueue","metric":0.0}

Shut down the ingress replicators (jive stop ingress-replicator).
Install the third search service 3 (search03.yourdomain.com). For instructions, see Installing One or More Search Servers.
Update the ingress replicator settings for the additional search node (as described below in Point to the New Search Node).

Then, there are a couple of ways to rebuild the index. Here are the options:

Option	Description
If the index is large (multiple gigabytes):	It's faster to copy the active search index from search service 1 or 2 to the new search service 3. You can find the active search index by looking in /usr/local/jive/services/search-service/main-args.properties file for CONTENT_SEARCH_HOME_DIRECTORY=var/data/contentSearch/. This property lists the location of the search indexes.
If the index is small (less than two gigabytes) or you do not care how long the rebuild takes:	First, complete steps 8-10; then start a rebuild from the Admin Console (Admin Console: System > Settings > Search and click Rebuild Index).

Start search service 3 (jive start search).
Add the third search node to your load balancer.
Finally, start up all of the ingress replicators and make sure all of the search services are running (jive start ingress-replicator).

Diagram of HA On-prem Search: How to add or remove a Search node

Point to the New Search Node

After you've successfully added the new node as described above, you'll need the other nodes to point to the new search service node. Here's how to do that:

In /usr/local/jive/services/ingress-replicator-service/, re-configure ingress replicator 1 and 2 to include search service 3. Set the load balancer to forward the ingress service to all machines holding the ingress replicator, and the ingress replicators to forward to all machines where the search service runs. The relevant parameters in the main-args.properties file should look something like this:
```
PORT=29000
REPLICATE_INGRESS_TO_THESE_COMMA_SEPARATED_HOST_COLON_PORT_TUPLES=guaranteed:all:search01.yourdomain.com:30000,guaranteed:all:search02.yourdomain.com:30000,guaranteed:all:search03.yourdomain.com:30000
REPLICATE_REBUILD_TO_THESE_COMMA_SEPARATED_HOST_COLON_PORT_TUPLES=search01.eng.yourdomain.com:30000,search02.eng.yourdomain.com:30000,search03.yourdomain.com:30000
REPLICATE_INDEX_MANAGE_TO_THESE_COMMA_SEPARATED_HOST_COLON_PORT_TUPLES=search01.eng.yourdomain.com:30000,search02.eng.yourdomain.com:30000
```
Restart the ingress replicator services (jive restart ingress-replicator).
Add the ingress replicator services back into the load balancer configuration.